Nov 03 06:56:19.072209 ip-10-0-129-24 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 06:56:19.072219 ip-10-0-129-24 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 06:56:19.072226 ip-10-0-129-24 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 06:56:19.072461 ip-10-0-129-24 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 06:56:29.293744 ip-10-0-129-24 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 06:56:29.293786 ip-10-0-129-24 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 230379fadd6d41c6bbd56f2683fc7338 --
Nov 03 06:58:25.579512 ip-10-0-129-24 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 06:58:25.937944 ip-10-0-129-24 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 06:58:25.937944 ip-10-0-129-24 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 06:58:25.937944 ip-10-0-129-24 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 06:58:25.937944 ip-10-0-129-24 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 06:58:25.943290 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943192    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:58:25.943290 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943289    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:58:25.943290 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943293    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943297    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943301    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943304    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943306    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943309    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943312    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943314    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943317    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943320    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943323    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943326    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943328    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943332    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943335    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943337    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943340    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943343    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943345    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943348    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:58:25.943383 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943351    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943353    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943356    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943358    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943360    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943363    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943365    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943368    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943371    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943380    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943383    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943385    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943388    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943391    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943394    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943397    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943399    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943401    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943404    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943406    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:58:25.943841 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943409    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943411    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943414    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943416    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943419    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943421    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943424    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943427    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943429    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943431    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943434    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943436    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943438    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943441    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943444    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943447    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943449    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943453    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943458    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:58:25.944331 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943461    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943464    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943467    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943470    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943472    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943475    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943478    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943480    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943482    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943485    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943487    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943490    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943492    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943495    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943497    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943507    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943512    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943515    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943518    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943520    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:58:25.944773 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943523    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943525    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943527    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943530    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943532    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943535    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943538    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943541    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943543    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943546    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943549    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943552    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943554    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943557    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943560    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943562    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943565    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943568    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943570    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943573    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:58:25.945251 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943575    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943578    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943580    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943583    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.943585    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945263    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945271    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945274    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945278    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945280    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945285    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945289    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945292    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945295    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945298    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945301    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945304    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945306    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945309    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:58:25.945725 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945311    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945314    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945316    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945319    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945322    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945325    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945327    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945330    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945332    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945335    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945338    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945340    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945343    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945345    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945348    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945351    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945354    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945357    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945359    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945362    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:58:25.946185 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945364    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945366    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945370    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945372    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945375    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945377    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945380    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945383    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945386    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945388    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945391    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945393    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945395    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945398    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945400    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945402    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945405    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945408    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945410    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945413    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:58:25.946661 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945415    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945418    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945421    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945423    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945426    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945429    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945431    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945434    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945437    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945439    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945442    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945444    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945447    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945449    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945451    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945454    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945457    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945459    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945462    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:58:25.947166 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945464    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945468    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945471    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945474    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945477    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945479    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945481    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945484    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945486    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945489    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945491    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945494    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945496    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945499    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945501    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945504    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945507    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945509    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945511    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:58:25.947646 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945514    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945516    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945519    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945521    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945524    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945526    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945530    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945532    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945534    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945537    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945540    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945542    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945545    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.945547    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946041    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946047    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946050    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946052    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946055    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:58:25.948137 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946058    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946061    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946063    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946070    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946073    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946076    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946078    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946081    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946084    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946087    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946089    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946092    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946095    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946097    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946099    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946102    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946104    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946107    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946109    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946112    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:58:25.948595 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946114    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946117    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946119    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946122    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946125    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946128    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946130    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946133    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946135    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946138    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946140    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946143    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946145    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946147    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946150    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946152    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946155    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946157    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946160    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:58:25.949068 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946162    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946165    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946168    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946172    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946175    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946178    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946181    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946183    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946186    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946188    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946191    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946208    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946211    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946213    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946216    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946218    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946221    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946224    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946227    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946229    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:58:25.949558 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946232    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946234    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946237    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946240    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946242    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946245    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946247    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946250    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946252    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946257    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946260    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946263    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946266    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946268    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946272    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946276    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946279    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946281    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946284    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:58:25.950030 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946286    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946289    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946291    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946294    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946296    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946298    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946301    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946303    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946306    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946309    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946311    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946314    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946317    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946319    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946321    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946324    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946326    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946329    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946331    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946333    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946336    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:58:25.950471 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946340    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.946344    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947170    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947180    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947186    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947191    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947207    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947211    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947215    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947220    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947223    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947226    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947229    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947233    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947236    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947239    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947242    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947245    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947248    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947251    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947257    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947260    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947263    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947268    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 06:58:25.950960 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947271    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947276    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947279    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947282    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947285    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947289    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947292    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947295    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947298    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947303    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947307    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947310    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947313    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947316    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947319    2576 flags.go:64] FLAG: --enable-server="true"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947321    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947326    2576 flags.go:64] FLAG: --event-burst="100"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947329    2576 flags.go:64] FLAG: --event-qps="50"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947332    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947335    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947338    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947342    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947346    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947349    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947352    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 03 06:58:25.951528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947354    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947358    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947361    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947363    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947366    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947369    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947372    2576 flags.go:64] FLAG: --feature-gates=""
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947377    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947380    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947383    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947385    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947388    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947391    2576 flags.go:64] FLAG: --help="false"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947394    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947397    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947400    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947405    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947408    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947411    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947414    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947417    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947420    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947422    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 06:58:25.952131 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947425    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947428    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947431    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947434    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947437    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947440    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947442    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947445    2576 flags.go:64] FLAG: --lock-file=""
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947448    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947450    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947453    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947459    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947461    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947464    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947467    2576 flags.go:64] FLAG: --logging-format="text"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947470    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947473    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947476    2576 flags.go:64] FLAG: --manifest-url=""
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947479    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947484    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947487    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947492    2576 flags.go:64] FLAG: --max-pods="110"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947495    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947498    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947501    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 06:58:25.952694 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947505    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947509    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947512    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947515    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947522    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947525    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947528    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947531    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947534    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947537    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947540    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947544    2576 flags.go:64] FLAG: --port="10250"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947546    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947549    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-021bc18b8e84ebaf3"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947552    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947555    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947558    2576 flags.go:64] FLAG: --register-node="true"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947561    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947565    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947568    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947570    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947573    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947577    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947580    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947583    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947587    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 06:58:25.953436 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947589    2576 flags.go:64] FLAG: --runonce="false"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947592    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947598    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947600    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947603    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947606    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947611    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947614    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947617    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947620    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947623    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947626    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947629    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947632    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947635    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947638    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947643    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947646    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947649    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947654    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947657    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947660    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947662    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947665    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947668    2576 flags.go:64] FLAG: --v="2"
Nov 03 06:58:25.954047 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947672    2576 flags.go:64] FLAG: --version="false"
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947676    2576 flags.go:64] FLAG: --vmodule=""
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947680    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.947684    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947861    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947866    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947870    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947873    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947876    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947881    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947884    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947887    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947889    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947893    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947896    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947898    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947901    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947904    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947906    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947909    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947911    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:58:25.954681 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947914    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947917    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947919    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947922    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947924    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947927    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947929    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947932    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947934    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947937    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947939    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947942    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947944    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947947    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947949    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947952    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947954    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947957    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947959    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:58:25.955171 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947962    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947966    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947968    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947970    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947973    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947976    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947978    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947981    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947983    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947986    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947989    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947991    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947993    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947996    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.947998    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948001    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948004    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948006    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948008    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948011    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:58:25.955659 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948014    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948017    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948019    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948022    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948025    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948029    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948032    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948034    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948037    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948039    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948042    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948045    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948047    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948055    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948059    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948061    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948064    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948066    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948069    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:58:25.956117 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948072    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948074    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948077    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948079    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948082    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948086    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948089    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948092    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948095    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948098    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948100    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948103    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948106    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948108    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948110    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948113    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948116    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948118    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948121    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:58:25.956588 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948123    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948126    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948128    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948131    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948134    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948136    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948139    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948142    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948146    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948148    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948151    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948154    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.948182    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948239    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948244    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948246    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:58:25.957041 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948249    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948251    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948254    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948256    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948259    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948262    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948264    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948267    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948269    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948272    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948274    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948277    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948279    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948282    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948284    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948287    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948289    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948293    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948296    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948298    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:58:25.957434 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948301    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948303    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948305    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948308    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948312    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948315    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948318    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948320    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948323    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948325    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948328    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948330    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948332    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948335    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948337    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948340    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948342    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948345    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948347    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:58:25.957917 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948350    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948352    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948355    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948357    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948360    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948362    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948365    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948367    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948369    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948372    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948375    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948378    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948380    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948382    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948385    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948389    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948392    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948396    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948399    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948402    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:58:25.958409 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948405    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948408    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948411    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948414    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948417    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948419    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948422    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948424    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948427    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948430    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948433    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948435    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948438    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948440    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948443    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948447    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948450    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948453    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948456    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:58:25.959087 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948459    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948461    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948464    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948468    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948471    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948473    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948475    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948478    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948480    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948483    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948487    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948489    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948492    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948495    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948498    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948501    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948504    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948506    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948509    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948511    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:58:25.959840 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948513    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948516    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948518    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948521    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.948523    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.955840    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.955861    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.955871    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.955876    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955931    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955936    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955939    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955942    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955945    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955948    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955950    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955953    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955956    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955959    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:58:25.960376 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955961    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955964    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955967    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955970    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955972    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955975    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955978    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955980    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955983    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955985    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955988    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955991    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955993    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955996    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.955998    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956001    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956003    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956006    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956009    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956012    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:58:25.960824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956015    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956017    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956020    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956022    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956025    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956027    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956030    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956032    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956035    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956038    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956040    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956042    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956045    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956047    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956050    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956052    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956054    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956057    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956060    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956063    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:58:25.961382 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956065    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956068    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956071    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956074    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956076    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956079    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956081    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956083    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956086    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956090    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956094    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956097    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956101    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956104    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956106    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956109    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956112    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956114    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956116    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:58:25.961854 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956119    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956122    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956124    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956127    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956130    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956132    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956135    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956137    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956140    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956143    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956146    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956149    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956151    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956154    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956157    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956159    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956162    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956165    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956169    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:58:25.962312 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956173    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956176    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956179    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956182    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956185    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956187    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956190    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956209    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956212    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956214    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956217    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956220    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956222    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956225    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956228    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956230    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956232    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:58:25.962768 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956235    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.956264    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956408    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956412    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956415    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956418    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956421    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956424    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956427    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956429    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956431    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956434    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956436    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956439    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956441    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956444    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:58:25.963167 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956447    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956449    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956451    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956454    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956456    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956459    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956461    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956464    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956467    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956469    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956472    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956474    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956476    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956479    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956481    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956484    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956486    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956489    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956491    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956494    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:58:25.963691 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956496    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956499    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956502    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956504    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956506    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956509    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956511    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956514    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956516    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956519    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956521    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956524    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956526    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956529    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956531    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956534    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956536    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956540    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956543    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:58:25.964143 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956546    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956551    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956555    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956558    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956561    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956563    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956566    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956568    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956570    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956573    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956576    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956578    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956581    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956583    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956586    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956588    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956591    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956594    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956596    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:58:25.964650 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956599    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956601    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956604    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956606    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956608    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956611    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956614    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956617    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956619    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956621    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956624    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956626    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956629    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956631    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956633    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956636    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956639    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956641    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956643    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:58:25.965092 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956646    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956648    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956651    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956653    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956655    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956658    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956660    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956662    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956665    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956668    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956670    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956673    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956675    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956678    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956680    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:58:25.965540 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.956704    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956739    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956742    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956747    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956750    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956753    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956756    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956758    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956761    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956763    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956766    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956768    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956771    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956773    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956776    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956779    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956781    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956784    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956786    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:58:25.965923 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956788    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956791    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956793    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956795    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956798    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956800    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956803    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956805    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956807    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956810    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956812    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956815    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956817    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956820    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956822    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956824    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956827    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956829    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956832    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:58:25.966422 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956834    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956837    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956839    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956842    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956844    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956846    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956849    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956851    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956853    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956856    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956859    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956861    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956864    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956866    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956868    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956871    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956873    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956876    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956878    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956881    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:58:25.966858 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956883    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956886    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956888    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956891    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956893    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956895    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956898    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956901    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956903    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956905    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956908    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956910    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956913    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956915    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956917    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956920    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956922    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956924    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956927    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956929    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:58:25.967342 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956931    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956935    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956939    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956942    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956946    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956950    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956952    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956956    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956958    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956961    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956963    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956965    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956971    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956974    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956976    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956979    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956981    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956984    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956987    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:58:25.967813 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956989    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956992    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956994    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956996    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.956999    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.957001    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.957004    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.957006    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.957009    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:25.957011    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.957068    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.961088    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.962152    2576 server.go:1014] "Starting client certificate rotation"
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.962227    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.962259    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.965010    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 06:58:25.968266 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.966154    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 06:58:25.980281 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.980189    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 03 06:58:25.984176 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.984163    2576 log.go:25] "Validated CRI v1 image API"
Nov 03 06:58:25.985050 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.985019    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 06:58:25.989208 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.989176    2576 fs.go:135] Filesystem UUIDs: map[1de08faf-a56b-48b8-9e52-49d5773e638d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9fbc50de-2a18-4d61-bf4c-9a46a3eae5b1:/dev/nvme0n1p4]
Nov 03 06:58:25.989274 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.989210    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 06:58:25.995063 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.994948    2576 manager.go:217] Machine: {Timestamp:2026-11-03 06:58:25.993709956 +0000 UTC m=+0.320138567 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103857 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d4197316f2349a84ccf421a7de303 SystemUUID:ec2d4197-316f-2349-a84c-cf421a7de303 BootID:230379fa-dd6d-41c6-bbd5-6f2683fc7338 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:01:98:a4:a1:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:01:98:a4:a1:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:ab:dc:ab:3a:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 06:58:25.995063 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.995060    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 06:58:25.995177 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.995145    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 06:58:25.996362 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.996334    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 06:58:25.996505 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.996364    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-24.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 06:58:25.996551 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.996512    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 06:58:25.996551 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.996520    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 06:58:25.996551 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.996531    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 06:58:25.997345 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.997335    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 06:58:25.998582 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:25.998573    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 06:58:26.003095 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.003084    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 06:58:26.003258 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.003249    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 06:58:26.004029 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.004020    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 06:58:26.004066 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.004034    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 06:58:26.004066 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.004044    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 06:58:26.004066 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.004053    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 03 06:58:26.004066 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.004061    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 06:58:26.004552 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.004523    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 06:58:26.008843 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.008824    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 06:58:26.015370 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.015356    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 06:58:26.016314 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.016303    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 06:58:26.016366 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.016333    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 06:58:26.016487 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.016477    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 06:58:26.016528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.016493    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 06:58:26.016528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.016500    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 06:58:26.016528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.016505    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 06:58:26.016528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.016511    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 06:58:26.016528 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.016516    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 06:58:26.017138 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.017130    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 06:58:26.017169 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.017139    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 06:58:26.017169 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.017146    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 06:58:26.017169 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.017152    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 06:58:26.017169 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.017160    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 06:58:26.017169 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.017169    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 06:58:26.017927 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.017918    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 06:58:26.017957 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.017928    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 06:58:26.021786 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.021773    2576 server.go:1263] "Started kubelet"
Nov 03 06:58:26.022041 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.021926    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 06:58:26.022168 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.021941    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 06:58:26.022239 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.022191    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 06:58:26.022527 ip-10-0-129-24 systemd[1]: Started Kubernetes Kubelet.
Nov 03 06:58:26.024179 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.024164    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 06:58:26.026433 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.026415    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 06:58:26.029576 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.029556    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 06:58:26.032135 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.032121    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 06:58:26.032222 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.032138    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 06:58:26.032310 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.032282    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found"
Nov 03 06:58:26.032919 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.032901    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 06:58:26.033054 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.033039    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 06:58:26.033373 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.033037    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 06:58:26.033373 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.033326    2576 factory.go:55] Registering systemd factory
Nov 03 06:58:26.033373 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.033389    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 03 06:58:26.033586 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.033455    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 06:58:26.033586 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.033464    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 06:58:26.033691 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.033626    2576 factory.go:153] Registering CRI-O factory
Nov 03 06:58:26.033691 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.033647    2576 factory.go:223] Registration of the crio container factory successfully
Nov 03 06:58:26.033764 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.033704    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 06:58:26.033764 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.033720    2576 factory.go:103] Registering Raw factory
Nov 03 06:58:26.033764 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.033731    2576 manager.go:1199] Started watching for new ooms in manager
Nov 03 06:58:26.034838 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.034822    2576 manager.go:319] Starting recovery of all containers
Nov 03 06:58:26.048425 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.048407    2576 manager.go:324] Recovery completed
Nov 03 06:58:26.052302 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.052287    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:58:26.054691 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.054675    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:58:26.054776 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.054707    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:58:26.054776 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.054720    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:58:26.055258 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.055241    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 03 06:58:26.055258 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.055255    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 06:58:26.055365 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.055260    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:58:26.055365 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.055275    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 06:58:26.056376 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.056363    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:58:26.056417 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.056387    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:58:26.056417 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.056398    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:58:26.057539 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.057527    2576 policy_none.go:50] "Start"
Nov 03 06:58:26.057577 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.057543    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 06:58:26.057577 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.057553    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 06:58:26.059510 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.059500    2576 policy_none.go:44] "Start"
Nov 03 06:58:26.092918 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.092754    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 06:58:26.097283 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.097264    2576 manager.go:341] "Starting Device Plugin manager"
Nov 03 06:58:26.097375 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.097313    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 06:58:26.097375 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.097327    2576 server.go:87] "Starting device plugin registration server"
Nov 03 06:58:26.097611 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.097595    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 06:58:26.097675 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.097615    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 06:58:26.097719 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.097709    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 06:58:26.097813 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.097800    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 06:58:26.097813 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.097810    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 06:58:26.098429 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.098406    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 06:58:26.098523 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.098450    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-24.us-west-1.compute.internal\" not found"
Nov 03 06:58:26.150149 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.150120    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 06:58:26.158610 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.158584    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 06:58:26.158715 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.158621    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 06:58:26.158715 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.158646    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 06:58:26.158715 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.158690    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 06:58:26.197984 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.197912    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:58:26.199243 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.199225    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:58:26.199345 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.199264    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:58:26.199345 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.199278    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:58:26.199345 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.199312    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.221613 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.221569    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-24.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 06:58:26.221613 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.221580    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-24.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 06:58:26.222282 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.221562    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-24.us-west-1.compute.internal.18e476bd11e9ec0e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-24.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-24.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 06:58:26.021739534 +0000 UTC m=+0.348168141,LastTimestamp:2026-11-03 06:58:26.021739534 +0000 UTC m=+0.348168141,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-24.us-west-1.compute.internal,}"
Nov 03 06:58:26.259285 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.259246    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"]
Nov 03 06:58:26.259423 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.259322    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:58:26.260319 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.260300    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:58:26.260427 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.260338    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:58:26.260427 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.260353    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:58:26.263017 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.263000    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:58:26.263170 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.263154    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.263239 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.263186    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:58:26.263858 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.263839    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:58:26.263939 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.263847    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:58:26.263939 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.263894    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:58:26.263939 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.263908    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:58:26.263939 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.263868    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:58:26.264060 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.263941    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:58:26.266028 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.266013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.266111 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.266035    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:58:26.268391 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.268363    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:58:26.268483 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.268410    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:58:26.268483 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.268427    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:58:26.290746 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.290727    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.292746 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.292728    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.316369 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.316352    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 06:58:26.316460 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.316398    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kr88l"
Nov 03 06:58:26.333724 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.333650    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-24.us-west-1.compute.internal.18e476bd13e0ba36  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-24.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-24.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-24.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 06:58:26.054691382 +0000 UTC m=+0.381119988,LastTimestamp:2026-11-03 06:58:26.054691382 +0000 UTC m=+0.381119988,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-24.us-west-1.compute.internal,}"
Nov 03 06:58:26.334260 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.334242    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 06:58:26.334333 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.334318    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.334390 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.334343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.334390 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.334360    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e6f5344ea2747ae378e95b2b7a4fbd6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"2e6f5344ea2747ae378e95b2b7a4fbd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.382164 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.382122    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 06:58:26.382164 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.382129    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-24.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 06:58:26.433306 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.433270    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 06:58:26.434842 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.434822    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-24.us-west-1.compute.internal\" not found" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.435270 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.435251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.435324 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.435281    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.435324 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.435300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e6f5344ea2747ae378e95b2b7a4fbd6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"2e6f5344ea2747ae378e95b2b7a4fbd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.435428 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.435356    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e6f5344ea2747ae378e95b2b7a4fbd6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"2e6f5344ea2747ae378e95b2b7a4fbd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.435428 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.435363    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.435428 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.435384    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.528077 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.528002    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-24.us-west-1.compute.internal\" not found" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.593327 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.593293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.596373 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.596348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.635718 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.635684    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:58:26.637001 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.636985    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:58:26.637063 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.637017    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:58:26.637063 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.637027    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:58:26.637063 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.637055    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.695025 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.694995    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:26.695025 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.695024    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-24.us-west-1.compute.internal\": node \"ip-10-0-129-24.us-west-1.compute.internal\" not found"
Nov 03 06:58:26.813331 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.813259    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found"
Nov 03 06:58:26.913855 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.913816    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found"
Nov 03 06:58:26.961563 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.961539    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 06:58:26.961970 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:26.961754    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 06:58:26.961970 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:26.961702    2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa5d097cf21004010b37820ad9a97310-f5a8f4c20a3cd5e4.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-24.us-west-1.compute.internal.18e476bd13e13812\": read tcp 10.0.129.24:41426->13.184.88.160:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-24.us-west-1.compute.internal.18e476bd13e13812  default   5081 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-24.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-24.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-24.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 06:58:26 +0000 UTC,LastTimestamp:2026-11-03 06:58:26.19928347 +0000 UTC m=+0.525712076,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-24.us-west-1.compute.internal,}"
Nov 03 06:58:27.013955 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:27.013929    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found"
Nov 03 06:58:27.114688 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:27.114623    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found"
Nov 03 06:58:27.215095 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:27.215064    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found"
Nov 03 06:58:27.267267 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.267243    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:58:27.268188 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.268174    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:58:27.268270 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.268222    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:58:27.268270 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.268232    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:58:27.315704 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:27.315676    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found"
Nov 03 06:58:27.320970 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.320947    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 06:53:27 +0000 UTC" deadline="2028-07-04 11:58:26.311137953 +0000 UTC"
Nov 03 06:58:27.321018 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.320971    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14620h59m58.990169721s"
Nov 03 06:58:27.384440 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.384376    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 06:58:27.433040 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.433014    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:27.481783 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.481764    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 06:58:27.497629 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.497606    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 06:58:27.498408 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.498395    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:58:27.557861 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.557831    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 06:58:27.663330 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:27.663290    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529279bc3624b3843e9992a4e421e883.slice/crio-1dd239fb9366ccf33b06de4b3aba8fd62060181960794ae44e434b68d3e23d6e WatchSource:0}: Error finding container 1dd239fb9366ccf33b06de4b3aba8fd62060181960794ae44e434b68d3e23d6e: Status 404 returned error can't find the container with id 1dd239fb9366ccf33b06de4b3aba8fd62060181960794ae44e434b68d3e23d6e
Nov 03 06:58:27.663856 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:27.663834    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6f5344ea2747ae378e95b2b7a4fbd6.slice/crio-a85e6df9a6ae53c3e7a1ad0586ef4fec6423d066c1fafba595a7514728697b17 WatchSource:0}: Error finding container a85e6df9a6ae53c3e7a1ad0586ef4fec6423d066c1fafba595a7514728697b17: Status 404 returned error can't find the container with id a85e6df9a6ae53c3e7a1ad0586ef4fec6423d066c1fafba595a7514728697b17
Nov 03 06:58:27.667952 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:27.667940    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 06:58:28.836904 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:28.836875    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 06:58:29.020370 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.020299    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 06:58:29.047409 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.047385    2576 apiserver.go:52] "Watching apiserver"
Nov 03 06:58:29.075554 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.075531    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 06:58:29.151344 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.151321    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 06:58:29.151636 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.151620    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-pw94d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7","openshift-multus/multus-additional-cni-plugins-xmqdp","openshift-network-operator/iptables-alerter-krd42","openshift-dns/node-resolver-dxw74","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal","openshift-multus/multus-2dp6g","openshift-network-diagnostics/network-check-target-6bhzs","kube-system/konnectivity-agent-9v5rw","openshift-ovn-kubernetes/ovnkube-node-6w8lk","kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-gqgl4","openshift-image-registry/node-ca-pkxfm","openshift-multus/network-metrics-daemon-mq4wx"]
Nov 03 06:58:29.155358 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.155342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.157854 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.157589    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:58:29.160536 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.160514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.162874 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.162732    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:58:29.164974 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.164958    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:58:29.167534 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.167516    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:29.167630 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.167606    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:29.169779 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.169762    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:29.171907 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.171891    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:29.171996 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.171953    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:29.174224 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.174207    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.176660 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.176647    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.178948 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.178937    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:29.179024 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.178999    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:29.181295 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.181281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.187849 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.187831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92fece83-25d4-4543-8489-67583d71ceff-iptables-alerter-script\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:58:29.187942 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.187856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-dbus\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:29.187942 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.187872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-systemd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.187942 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.187886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-script-lib\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.187942 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.187901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-k8s-cni-cncf-io\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.188114 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.187956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-hostroot\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.188114 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c46b893-7db9-4a89-a701-12fa631394fa-hosts-file\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:58:29.188114 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-etc-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.188114 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188106    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scr95\" (UniqueName: \"kubernetes.io/projected/021ef7d9-672a-4953-bb37-3ca4c271c7ca-kube-api-access-scr95\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.188327 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:29.188327 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-registration-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.188327 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188256    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-netns\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.188327 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-bin\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.188474 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-netns\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.188474 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188374    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-bin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.188474 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovn-node-metrics-cert\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.188474 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-system-cni-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.188474 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.188622 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.188622 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188540    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-kubelet\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.188622 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn9q6\" (UniqueName: \"kubernetes.io/projected/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-kube-api-access-jn9q6\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.188622 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-konnectivity-ca\") pod \"konnectivity-agent-9v5rw\" (UID: \"b5aebe2f-412e-4d2a-b644-7fd13bb62e07\") " pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:29.188751 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-ovn\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.188751 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-daemon-config\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.188751 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.188751 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c46b893-7db9-4a89-a701-12fa631394fa-tmp-dir\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:58:29.188751 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kslf\" (UniqueName: \"kubernetes.io/projected/9c46b893-7db9-4a89-a701-12fa631394fa-kube-api-access-6kslf\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:58:29.188751 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-agent-certs\") pod \"konnectivity-agent-9v5rw\" (UID: \"b5aebe2f-412e-4d2a-b644-7fd13bb62e07\") " pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:29.188965 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.188965 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-node-log\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.188965 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-log-socket\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.188965 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-kubelet-config\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:29.188965 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188842    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-socket-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.188965 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-netd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.188965 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-kubelet\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.188965 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.189289 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.188989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b46q\" (UniqueName: \"kubernetes.io/projected/92fece83-25d4-4543-8489-67583d71ceff-kube-api-access-2b46q\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:58:29.189289 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.189289 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-system-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.189289 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-multus\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.189289 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7bgl\" (UniqueName: \"kubernetes.io/projected/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-kube-api-access-n7bgl\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.189289 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/017d594a-16ee-456f-8b9f-2fecc89acb9b-host\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:58:29.189289 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189216    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-var-lib-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.189289 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cni-binary-copy\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.189289 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cnibin\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.189289 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-systemd-units\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-config\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-multus-certs\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-device-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189495    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92fece83-25d4-4543-8489-67583d71ceff-host-slash\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-slash\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cnibin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189642    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgtp\" (UniqueName: \"kubernetes.io/projected/017d594a-16ee-456f-8b9f-2fecc89acb9b-kube-api-access-9zgtp\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:58:29.189689 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-etc-kubernetes\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.190073 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9lfh\" (UniqueName: \"kubernetes.io/projected/7d04cba6-64ad-4a4e-8045-3c91d3237a14-kube-api-access-r9lfh\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:29.190073 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-env-overrides\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.190073 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-conf-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.190073 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/017d594a-16ee-456f-8b9f-2fecc89acb9b-serviceca\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:58:29.190073 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-sys-fs\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.190073 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljw5\" (UniqueName: \"kubernetes.io/projected/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kube-api-access-bljw5\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.190073 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.189905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-socket-dir-parent\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.205392 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.205353    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal" event={"ID":"2e6f5344ea2747ae378e95b2b7a4fbd6","Type":"ContainerStarted","Data":"a85e6df9a6ae53c3e7a1ad0586ef4fec6423d066c1fafba595a7514728697b17"}
Nov 03 06:58:29.206458 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.206437    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal" event={"ID":"529279bc3624b3843e9992a4e421e883","Type":"ContainerStarted","Data":"1dd239fb9366ccf33b06de4b3aba8fd62060181960794ae44e434b68d3e23d6e"}
Nov 03 06:58:29.210462 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.210438    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 06:58:29.210462 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.210438    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 06:58:29.210560 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.210512    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 06:58:29.210595 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.210563    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 06:58:29.210756 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.210743    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 06:58:29.210990 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.210977    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5znfn\""
Nov 03 06:58:29.211030 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.211001    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 06:58:29.211111 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.211099    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-htlhr\""
Nov 03 06:58:29.211358 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.211341    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 06:58:29.213635 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.213620    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 06:58:29.213746 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.213733    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 06:58:29.213798 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.213770    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44mbd\""
Nov 03 06:58:29.216041 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.216029    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lr4sn\""
Nov 03 06:58:29.216107 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.216099    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 06:58:29.216315 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.216299    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 06:58:29.216355 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.216307    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 06:58:29.218321 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.218305    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 06:58:29.218357 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.218329    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 06:58:29.218720 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.218709    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8c889\""
Nov 03 06:58:29.223275 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.223045    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 06:58:29.223843 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.223826    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxfbc\""
Nov 03 06:58:29.225066 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.223991    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 06:58:29.232775 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.232752    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 06:58:29.232775 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.232771    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 06:58:29.232885 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.232785    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l9lht\""
Nov 03 06:58:29.232934 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.232922    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 06:58:29.234908 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.234894    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 06:58:29.235166 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.235152    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-br79x\""
Nov 03 06:58:29.235234 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.235186    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 06:58:29.235234 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.235189    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 06:58:29.235234 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.235231    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 06:58:29.235341 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.235263    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 06:58:29.254915 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.254899    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 06:58:29.258679 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.258664    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 06:58:29.258755 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.258713    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 06:58:29.258755 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.258727    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dmgt\""
Nov 03 06:58:29.258755 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.258736    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 06:58:29.258866 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.258781    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gbgkc"
Nov 03 06:58:29.275985 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.275939    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 06:58:29.290310 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c46b893-7db9-4a89-a701-12fa631394fa-tmp-dir\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:58:29.290385 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kslf\" (UniqueName: \"kubernetes.io/projected/9c46b893-7db9-4a89-a701-12fa631394fa-kube-api-access-6kslf\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:58:29.290385 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-agent-certs\") pod \"konnectivity-agent-9v5rw\" (UID: \"b5aebe2f-412e-4d2a-b644-7fd13bb62e07\") " pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:29.290483 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.290483 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290425    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c46b893-7db9-4a89-a701-12fa631394fa-tmp-dir\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:58:29.290483 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290472    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.290622 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-node-log\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.290622 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-log-socket\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.290622 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-kubelet-config\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:29.290768 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-socket-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.290768 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290630    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-node-log\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.290768 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290651    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-log-socket\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.290768 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290664    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-kubelet-config\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:29.290768 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290689    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-netd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.290768 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-kubelet\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.290768 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290720    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-socket-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.290768 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.290768 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b46q\" (UniqueName: \"kubernetes.io/projected/92fece83-25d4-4543-8489-67583d71ceff-kube-api-access-2b46q\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290770    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-netd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-run\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290801    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-kubelet\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290783    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-system-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290883    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-multus\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290961    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-multus\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290964    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-system-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.290968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7bgl\" (UniqueName: \"kubernetes.io/projected/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-kube-api-access-n7bgl\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysconfig\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291033    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/017d594a-16ee-456f-8b9f-2fecc89acb9b-host\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-var-lib-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cni-binary-copy\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cnibin\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.291174 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291094    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/017d594a-16ee-456f-8b9f-2fecc89acb9b-host\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291126    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-var-lib-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-systemd-units\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291172    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cnibin\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291177    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-config\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291218    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291243    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291257    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-systemd-units\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-multus-certs\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-var-lib-kubelet\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-device-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.291271    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.291372    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs podName:7d04cba6-64ad-4a4e-8045-3c91d3237a14 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:29.791355606 +0000 UTC m=+3.075849861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs") pod "network-metrics-daemon-mq4wx" (UID: "7d04cba6-64ad-4a4e-8045-3c91d3237a14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:29.291882 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92fece83-25d4-4543-8489-67583d71ceff-host-slash\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-modprobe-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291435    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-systemd\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-slash\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291506    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cnibin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291520    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-device-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgtp\" (UniqueName: \"kubernetes.io/projected/017d594a-16ee-456f-8b9f-2fecc89acb9b-kube-api-access-9zgtp\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291315    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-multus-certs\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291586    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-slash\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291608    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92fece83-25d4-4543-8489-67583d71ceff-host-slash\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-etc-kubernetes\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291643    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291645    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cnibin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9lfh\" (UniqueName: \"kubernetes.io/projected/7d04cba6-64ad-4a4e-8045-3c91d3237a14-kube-api-access-r9lfh\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291698    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-etc-kubernetes\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-lib-modules\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291839    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-host\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.292494 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-env-overrides\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.291981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d5mj\" (UniqueName: \"kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj\") pod \"network-check-target-6bhzs\" (UID: \"af5953b4-e51d-433b-9860-6b077061f77c\") " pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-tuned\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-conf-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/017d594a-16ee-456f-8b9f-2fecc89acb9b-serviceca\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-sys-fs\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292128    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-conf-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljw5\" (UniqueName: \"kubernetes.io/projected/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kube-api-access-bljw5\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292201    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-sys-fs\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-socket-dir-parent\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292312    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cni-binary-copy\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292370    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92fece83-25d4-4543-8489-67583d71ceff-iptables-alerter-script\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292388    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-socket-dir-parent\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8vm\" (UniqueName: \"kubernetes.io/projected/5cea0af3-b692-43ed-8c86-df442fd5378c-kube-api-access-bx8vm\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292494    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-config\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-dbus\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292587    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/017d594a-16ee-456f-8b9f-2fecc89acb9b-serviceca\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:58:29.293279 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292605    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-systemd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292647    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-systemd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292656    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-dbus\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-script-lib\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-k8s-cni-cncf-io\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292754    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-k8s-cni-cncf-io\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-hostroot\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292815    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-env-overrides\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292822    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92fece83-25d4-4543-8489-67583d71ceff-iptables-alerter-script\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-conf\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292862    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-hostroot\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c46b893-7db9-4a89-a701-12fa631394fa-hosts-file\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-etc-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scr95\" (UniqueName: \"kubernetes.io/projected/021ef7d9-672a-4953-bb37-3ca4c271c7ca-kube-api-access-scr95\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292963    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c46b893-7db9-4a89-a701-12fa631394fa-hosts-file\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292974    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-etc-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.292990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-sys\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.293992 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293077    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-registration-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-netns\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.293182    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-bin\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293197    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-registration-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.293227    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret podName:cf458915-e617-486b-8b9f-9019d09ca398 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:29.793211117 +0000 UTC m=+3.077705358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret") pod "global-pull-secret-syncer-pw94d" (UID: "cf458915-e617-486b-8b9f-9019d09ca398") : object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-netns\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293256    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-script-lib\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293266    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-bin\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293224    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-netns\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-bin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293310    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-netns\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293335    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-bin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovn-node-metrics-cert\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-system-cni-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.294633 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-kubernetes\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293441    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-system-cni-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-kubelet\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9q6\" (UniqueName: \"kubernetes.io/projected/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-kube-api-access-jn9q6\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293563    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293588    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-kubelet\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-tmp\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-konnectivity-ca\") pod \"konnectivity-agent-9v5rw\" (UID: \"b5aebe2f-412e-4d2a-b644-7fd13bb62e07\") " pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-ovn\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293869    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-daemon-config\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.293905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-ovn\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.294468    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.295122 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.294702    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-agent-certs\") pod \"konnectivity-agent-9v5rw\" (UID: \"b5aebe2f-412e-4d2a-b644-7fd13bb62e07\") " pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:29.295603 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.295222    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.295603 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.295277    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-konnectivity-ca\") pod \"konnectivity-agent-9v5rw\" (UID: \"b5aebe2f-412e-4d2a-b644-7fd13bb62e07\") " pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:29.295603 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.295310    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-daemon-config\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.295688 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.295628    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovn-node-metrics-cert\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.356000 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.355968    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b46q\" (UniqueName: \"kubernetes.io/projected/92fece83-25d4-4543-8489-67583d71ceff-kube-api-access-2b46q\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:58:29.356125 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.355976    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7bgl\" (UniqueName: \"kubernetes.io/projected/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-kube-api-access-n7bgl\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.356620 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.356463    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kslf\" (UniqueName: \"kubernetes.io/projected/9c46b893-7db9-4a89-a701-12fa631394fa-kube-api-access-6kslf\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:58:29.356620 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.356480    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgtp\" (UniqueName: \"kubernetes.io/projected/017d594a-16ee-456f-8b9f-2fecc89acb9b-kube-api-access-9zgtp\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:58:29.356620 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.356564    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scr95\" (UniqueName: \"kubernetes.io/projected/021ef7d9-672a-4953-bb37-3ca4c271c7ca-kube-api-access-scr95\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.356824 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.356783    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9q6\" (UniqueName: \"kubernetes.io/projected/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-kube-api-access-jn9q6\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.356902 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.356884    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9lfh\" (UniqueName: \"kubernetes.io/projected/7d04cba6-64ad-4a4e-8045-3c91d3237a14-kube-api-access-r9lfh\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:29.357038 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.357020    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljw5\" (UniqueName: \"kubernetes.io/projected/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kube-api-access-bljw5\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.394736 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-var-lib-kubelet\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.394853 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394742    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-modprobe-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.394853 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-systemd\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.394853 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-lib-modules\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.394853 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-host\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.394853 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5mj\" (UniqueName: \"kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj\") pod \"network-check-target-6bhzs\" (UID: \"af5953b4-e51d-433b-9860-6b077061f77c\") " pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:29.394853 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-tuned\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.394853 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8vm\" (UniqueName: \"kubernetes.io/projected/5cea0af3-b692-43ed-8c86-df442fd5378c-kube-api-access-bx8vm\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394877    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-host\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394855    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-var-lib-kubelet\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394901    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-systemd\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394900    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-modprobe-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394913    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-lib-modules\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-conf\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.394967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-sys\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.395008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-kubernetes\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.395033    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.395038    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-conf\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.395057    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-tmp\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.395075    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-sys\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.395090    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-run\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.395108    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-kubernetes\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.395114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysconfig\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.395175    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-run\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395180 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.395184    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysconfig\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.395686 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.395202    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.396982 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.396959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-tuned\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.397072 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.397014    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-tmp\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.455490 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.455468    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 06:58:29.455490 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.455488    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 06:58:29.455622 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.455500    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2d5mj for pod openshift-network-diagnostics/network-check-target-6bhzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:29.455622 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.455562    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj podName:af5953b4-e51d-433b-9860-6b077061f77c nodeName:}" failed. No retries permitted until 2026-11-03 06:58:29.955546846 +0000 UTC m=+3.240041097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2d5mj" (UniqueName: "kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj") pod "network-check-target-6bhzs" (UID: "af5953b4-e51d-433b-9860-6b077061f77c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:29.457544 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.457513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8vm\" (UniqueName: \"kubernetes.io/projected/5cea0af3-b692-43ed-8c86-df442fd5378c-kube-api-access-bx8vm\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.467435 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.467415    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2dp6g"
Nov 03 06:58:29.473814 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.473793    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:58:29.477793 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:29.477772    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021ef7d9_672a_4953_bb37_3ca4c271c7ca.slice/crio-8aedb969ece40c2805d597a3c2bb1334bb716722787f1e34469c68ea283b7704 WatchSource:0}: Error finding container 8aedb969ece40c2805d597a3c2bb1334bb716722787f1e34469c68ea283b7704: Status 404 returned error can't find the container with id 8aedb969ece40c2805d597a3c2bb1334bb716722787f1e34469c68ea283b7704
Nov 03 06:58:29.482061 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:29.482040    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017d594a_16ee_456f_8b9f_2fecc89acb9b.slice/crio-b4aa3efb81e74ecbdd6da5dc69c924b7233bd5c6c72c913692f3f040eed58b17 WatchSource:0}: Error finding container b4aa3efb81e74ecbdd6da5dc69c924b7233bd5c6c72c913692f3f040eed58b17: Status 404 returned error can't find the container with id b4aa3efb81e74ecbdd6da5dc69c924b7233bd5c6c72c913692f3f040eed58b17
Nov 03 06:58:29.490183 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.490162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:58:29.491221 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.491204    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:58:29.494852 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.494834    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:58:29.497089 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.496994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:29.499549 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:29.499526    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fece83_25d4_4543_8489_67583d71ceff.slice/crio-4b320f9d0777a26fea988bb6b747043a8bb056d60fb4a68eb9f07ccb50eee367 WatchSource:0}: Error finding container 4b320f9d0777a26fea988bb6b747043a8bb056d60fb4a68eb9f07ccb50eee367: Status 404 returned error can't find the container with id 4b320f9d0777a26fea988bb6b747043a8bb056d60fb4a68eb9f07ccb50eee367
Nov 03 06:58:29.502406 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:29.502381    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c46b893_7db9_4a89_a701_12fa631394fa.slice/crio-2826d0dc2418bdd741086e4fd1a1c5a293213e4647d24d5000c47799a1098e54 WatchSource:0}: Error finding container 2826d0dc2418bdd741086e4fd1a1c5a293213e4647d24d5000c47799a1098e54: Status 404 returned error can't find the container with id 2826d0dc2418bdd741086e4fd1a1c5a293213e4647d24d5000c47799a1098e54
Nov 03 06:58:29.504571 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.504553    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:58:29.506774 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:29.506749    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ab7be9_13bf_4798_bcb0_44ca9923ce0c.slice/crio-c62863da801620bda30b9d21be344d93340900bb64fbc9bdaa8db16212f82e31 WatchSource:0}: Error finding container c62863da801620bda30b9d21be344d93340900bb64fbc9bdaa8db16212f82e31: Status 404 returned error can't find the container with id c62863da801620bda30b9d21be344d93340900bb64fbc9bdaa8db16212f82e31
Nov 03 06:58:29.507825 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:29.507776    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5aebe2f_412e_4d2a_b644_7fd13bb62e07.slice/crio-14ea2886b367f03e10c8a977483adc77764c9d22c5908dcfd1b920fe13019cfd WatchSource:0}: Error finding container 14ea2886b367f03e10c8a977483adc77764c9d22c5908dcfd1b920fe13019cfd: Status 404 returned error can't find the container with id 14ea2886b367f03e10c8a977483adc77764c9d22c5908dcfd1b920fe13019cfd
Nov 03 06:58:29.513263 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:29.513243    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31cdbad_0b70_43c3_89c8_3fce65ba4ec3.slice/crio-71216bcbfdc6bb3860968ef41283611d25b43b64dfb52a25eaf5e258e251bfcf WatchSource:0}: Error finding container 71216bcbfdc6bb3860968ef41283611d25b43b64dfb52a25eaf5e258e251bfcf: Status 404 returned error can't find the container with id 71216bcbfdc6bb3860968ef41283611d25b43b64dfb52a25eaf5e258e251bfcf
Nov 03 06:58:29.517287 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.517267    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:58:29.523169 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.523154    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:29.532514 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:29.532494    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcf499f_e3d7_4146_89f8_74e2ae1685e3.slice/crio-b8a36f1128dcc6662802931f6d8de793885e2bb329f1b262eb6e81bb450f85f2 WatchSource:0}: Error finding container b8a36f1128dcc6662802931f6d8de793885e2bb329f1b262eb6e81bb450f85f2: Status 404 returned error can't find the container with id b8a36f1128dcc6662802931f6d8de793885e2bb329f1b262eb6e81bb450f85f2
Nov 03 06:58:29.798270 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.798186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:29.798270 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.798247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:29.798549 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.798352    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:29.798549 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.798404    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret podName:cf458915-e617-486b-8b9f-9019d09ca398 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:30.798386386 +0000 UTC m=+4.082880641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret") pod "global-pull-secret-syncer-pw94d" (UID: "cf458915-e617-486b-8b9f-9019d09ca398") : object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:29.798549 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.798461    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:29.798549 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.798493    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs podName:7d04cba6-64ad-4a4e-8045-3c91d3237a14 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:30.798483011 +0000 UTC m=+4.082977249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs") pod "network-metrics-daemon-mq4wx" (UID: "7d04cba6-64ad-4a4e-8045-3c91d3237a14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:29.999151 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:29.999100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5mj\" (UniqueName: \"kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj\") pod \"network-check-target-6bhzs\" (UID: \"af5953b4-e51d-433b-9860-6b077061f77c\") " pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:29.999335 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.999295    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 06:58:29.999335 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.999319    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 06:58:29.999335 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.999333    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2d5mj for pod openshift-network-diagnostics/network-check-target-6bhzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:29.999483 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:29.999396    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj podName:af5953b4-e51d-433b-9860-6b077061f77c nodeName:}" failed. No retries permitted until 2026-11-03 06:58:30.999378437 +0000 UTC m=+4.283872692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2d5mj" (UniqueName: "kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj") pod "network-check-target-6bhzs" (UID: "af5953b4-e51d-433b-9860-6b077061f77c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:30.209584 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.209506    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk" event={"ID":"7bcf499f-e3d7-4146-89f8-74e2ae1685e3","Type":"ContainerStarted","Data":"b8a36f1128dcc6662802931f6d8de793885e2bb329f1b262eb6e81bb450f85f2"}
Nov 03 06:58:30.210949 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.210924    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmqdp" event={"ID":"01ab7be9-13bf-4798-bcb0-44ca9923ce0c","Type":"ContainerStarted","Data":"c62863da801620bda30b9d21be344d93340900bb64fbc9bdaa8db16212f82e31"}
Nov 03 06:58:30.212350 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.212317    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dxw74" event={"ID":"9c46b893-7db9-4a89-a701-12fa631394fa","Type":"ContainerStarted","Data":"2826d0dc2418bdd741086e4fd1a1c5a293213e4647d24d5000c47799a1098e54"}
Nov 03 06:58:30.213542 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.213516    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pkxfm" event={"ID":"017d594a-16ee-456f-8b9f-2fecc89acb9b","Type":"ContainerStarted","Data":"b4aa3efb81e74ecbdd6da5dc69c924b7233bd5c6c72c913692f3f040eed58b17"}
Nov 03 06:58:30.214637 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.214616    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gqgl4" event={"ID":"5cea0af3-b692-43ed-8c86-df442fd5378c","Type":"ContainerStarted","Data":"ac3daa26b46f9b14eccdcbdc1128c37599af48fb0f3a518fc819241c5f7332e2"}
Nov 03 06:58:30.216157 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.216024    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7" event={"ID":"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3","Type":"ContainerStarted","Data":"71216bcbfdc6bb3860968ef41283611d25b43b64dfb52a25eaf5e258e251bfcf"}
Nov 03 06:58:30.217529 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.217474    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9v5rw" event={"ID":"b5aebe2f-412e-4d2a-b644-7fd13bb62e07","Type":"ContainerStarted","Data":"14ea2886b367f03e10c8a977483adc77764c9d22c5908dcfd1b920fe13019cfd"}
Nov 03 06:58:30.218760 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.218679    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-krd42" event={"ID":"92fece83-25d4-4543-8489-67583d71ceff","Type":"ContainerStarted","Data":"4b320f9d0777a26fea988bb6b747043a8bb056d60fb4a68eb9f07ccb50eee367"}
Nov 03 06:58:30.219922 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.219896    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2dp6g" event={"ID":"021ef7d9-672a-4953-bb37-3ca4c271c7ca","Type":"ContainerStarted","Data":"8aedb969ece40c2805d597a3c2bb1334bb716722787f1e34469c68ea283b7704"}
Nov 03 06:58:30.261904 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.261832    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 06:53:29 +0000 UTC" deadline="2028-08-21 05:19:31.944831562 +0000 UTC"
Nov 03 06:58:30.261904 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.261899    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15766h21m1.682936433s"
Nov 03 06:58:30.806032 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.805993    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:30.806284 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:30.806072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:30.806399 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:30.806315    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:30.806452 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:30.806443    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs podName:7d04cba6-64ad-4a4e-8045-3c91d3237a14 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:32.806399828 +0000 UTC m=+6.090894079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs") pod "network-metrics-daemon-mq4wx" (UID: "7d04cba6-64ad-4a4e-8045-3c91d3237a14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:30.806549 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:30.806534    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:30.806549 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:30.806579    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret podName:cf458915-e617-486b-8b9f-9019d09ca398 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:32.806567501 +0000 UTC m=+6.091061742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret") pod "global-pull-secret-syncer-pw94d" (UID: "cf458915-e617-486b-8b9f-9019d09ca398") : object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:31.009564 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:31.009532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5mj\" (UniqueName: \"kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj\") pod \"network-check-target-6bhzs\" (UID: \"af5953b4-e51d-433b-9860-6b077061f77c\") " pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:31.009740 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:31.009705    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 06:58:31.009740 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:31.009722    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 06:58:31.009740 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:31.009732    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2d5mj for pod openshift-network-diagnostics/network-check-target-6bhzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:31.009900 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:31.009782    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj podName:af5953b4-e51d-433b-9860-6b077061f77c nodeName:}" failed. No retries permitted until 2026-11-03 06:58:33.009769013 +0000 UTC m=+6.294263265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2d5mj" (UniqueName: "kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj") pod "network-check-target-6bhzs" (UID: "af5953b4-e51d-433b-9860-6b077061f77c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:31.203537 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:31.203505    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:31.203684 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:31.203648    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:31.203754 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:31.203738    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:31.203848 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:31.203830    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:31.203913 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:31.203899    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:31.203979 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:31.203962    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:31.263051 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:31.263010    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 06:53:29 +0000 UTC" deadline="2028-05-20 05:10:13.862924894 +0000 UTC"
Nov 03 06:58:31.263051 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:31.263048    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13534h11m42.59988128s"
Nov 03 06:58:32.238160 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:32.237904    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal" event={"ID":"2e6f5344ea2747ae378e95b2b7a4fbd6","Type":"ContainerStarted","Data":"d35b0d980303afb0a6fd3a68f6b854ddd056b00f4831f7aac4a51980b9f2c97b"}
Nov 03 06:58:32.245092 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:32.245058    2576 generic.go:356] "Generic (PLEG): container finished" podID="529279bc3624b3843e9992a4e421e883" containerID="625a8dcd2ee4a3a0750f65eab1b256794dac1d86a8dc048d2726c0dc3ee050e9" exitCode=0
Nov 03 06:58:32.245252 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:32.245103    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal" event={"ID":"529279bc3624b3843e9992a4e421e883","Type":"ContainerDied","Data":"625a8dcd2ee4a3a0750f65eab1b256794dac1d86a8dc048d2726c0dc3ee050e9"}
Nov 03 06:58:32.486159 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:32.486075    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal" podStartSLOduration=4.486057372 podStartE2EDuration="4.486057372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:32.358066676 +0000 UTC m=+5.642560937" watchObservedRunningTime="2026-11-03 06:58:32.486057372 +0000 UTC m=+5.770551635"
Nov 03 06:58:32.829079 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:32.828647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:32.829079 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:32.828724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:32.829079 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:32.828867    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:32.829079 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:32.828927    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs podName:7d04cba6-64ad-4a4e-8045-3c91d3237a14 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:36.828910151 +0000 UTC m=+10.113404393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs") pod "network-metrics-daemon-mq4wx" (UID: "7d04cba6-64ad-4a4e-8045-3c91d3237a14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:32.829079 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:32.828993    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:32.829079 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:32.829025    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret podName:cf458915-e617-486b-8b9f-9019d09ca398 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:36.829015259 +0000 UTC m=+10.113509501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret") pod "global-pull-secret-syncer-pw94d" (UID: "cf458915-e617-486b-8b9f-9019d09ca398") : object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:33.030391 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:33.030355    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5mj\" (UniqueName: \"kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj\") pod \"network-check-target-6bhzs\" (UID: \"af5953b4-e51d-433b-9860-6b077061f77c\") " pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:33.030627 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:33.030526    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 06:58:33.030627 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:33.030615    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 06:58:33.030627 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:33.030629    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2d5mj for pod openshift-network-diagnostics/network-check-target-6bhzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:33.030808 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:33.030696    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj podName:af5953b4-e51d-433b-9860-6b077061f77c nodeName:}" failed. No retries permitted until 2026-11-03 06:58:37.03068005 +0000 UTC m=+10.315174294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2d5mj" (UniqueName: "kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj") pod "network-check-target-6bhzs" (UID: "af5953b4-e51d-433b-9860-6b077061f77c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:33.210053 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:33.209975    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:33.210221 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:33.210099    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:33.210221 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:33.210191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:33.210337 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:33.210270    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:33.210337 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:33.210324    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:33.210440 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:33.210387    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:33.254127 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:33.254073    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal" event={"ID":"529279bc3624b3843e9992a4e421e883","Type":"ContainerStarted","Data":"230a7af561897fa89adfdaf79248eb030d57d740f38bd90efd930119f32fe5b7"}
Nov 03 06:58:33.263482 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:33.262554    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="4ab8753f067605fe0e64790646f78bd3ef9791974330de4fd81e64fd0acac7a8" exitCode=0
Nov 03 06:58:33.263482 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:33.263418    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmqdp" event={"ID":"01ab7be9-13bf-4798-bcb0-44ca9923ce0c","Type":"ContainerDied","Data":"4ab8753f067605fe0e64790646f78bd3ef9791974330de4fd81e64fd0acac7a8"}
Nov 03 06:58:33.498407 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:33.498299    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal" podStartSLOduration=5.498276187 podStartE2EDuration="5.498276187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:33.372652141 +0000 UTC m=+6.657146412" watchObservedRunningTime="2026-11-03 06:58:33.498276187 +0000 UTC m=+6.782770447"
Nov 03 06:58:34.270837 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:34.270754    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9v5rw" event={"ID":"b5aebe2f-412e-4d2a-b644-7fd13bb62e07","Type":"ContainerStarted","Data":"56fa22351ea6c0e828555872149073f3e0087e574bdc684721f6d14b3979207f"}
Nov 03 06:58:35.201518 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:35.201429    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:35.201971 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:35.201564    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:35.201971 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:35.201949    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:35.202077 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:35.202055    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:35.202128 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:35.202117    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:35.202271 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:35.202207    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:35.277827 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:35.277786    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pkxfm" event={"ID":"017d594a-16ee-456f-8b9f-2fecc89acb9b","Type":"ContainerStarted","Data":"9320a48be17b53e90987c1a17713eecf08eab995ac9424c5758c3d6cca9883c8"}
Nov 03 06:58:35.396684 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:35.396515    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9v5rw" podStartSLOduration=4.241916731 podStartE2EDuration="8.396495528s" totalImagesPullingTime="4.154578797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:34.392322332 +0000 UTC m=+7.676816595" watchObservedRunningTime="2026-11-03 06:58:35.396495528 +0000 UTC m=+8.680989790"
Nov 03 06:58:36.280819 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:36.280787    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7" event={"ID":"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3","Type":"ContainerStarted","Data":"7214831d948e3e1372d9bb73086a9f0f790d05cd9d80548c6818378c80d64af0"}
Nov 03 06:58:36.864502 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:36.863908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:36.864502 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:36.863955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:36.864502 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:36.864045    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:36.864502 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:36.864091    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs podName:7d04cba6-64ad-4a4e-8045-3c91d3237a14 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:44.864077836 +0000 UTC m=+18.148572074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs") pod "network-metrics-daemon-mq4wx" (UID: "7d04cba6-64ad-4a4e-8045-3c91d3237a14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:36.864502 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:36.864420    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:36.864502 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:36.864455    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret podName:cf458915-e617-486b-8b9f-9019d09ca398 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:44.864445976 +0000 UTC m=+18.148940215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret") pod "global-pull-secret-syncer-pw94d" (UID: "cf458915-e617-486b-8b9f-9019d09ca398") : object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:37.066071 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:37.066024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5mj\" (UniqueName: \"kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj\") pod \"network-check-target-6bhzs\" (UID: \"af5953b4-e51d-433b-9860-6b077061f77c\") " pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:37.066273 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:37.066235    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 06:58:37.066273 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:37.066256    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 06:58:37.066273 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:37.066267    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2d5mj for pod openshift-network-diagnostics/network-check-target-6bhzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:37.066440 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:37.066307    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj podName:af5953b4-e51d-433b-9860-6b077061f77c nodeName:}" failed. No retries permitted until 2026-11-03 06:58:45.066296458 +0000 UTC m=+18.350790696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2d5mj" (UniqueName: "kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj") pod "network-check-target-6bhzs" (UID: "af5953b4-e51d-433b-9860-6b077061f77c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:37.207083 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:37.206340    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:37.207083 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:37.206458    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:37.207083 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:37.206521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:37.207083 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:37.206600    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:37.207083 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:37.206637    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:37.207083 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:37.206704    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:37.824154 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:37.824082    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 06:58:38.146314 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:38.146203    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T06:58:37.824103435Z","UUID":"55c3f809-b7e7-4031-b5e3-fb3c543b94f7","Handler":null,"Name":"","Endpoint":""}
Nov 03 06:58:38.147627 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:38.147596    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 06:58:38.147627 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:38.147628    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 06:58:38.286662 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:38.286575    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmqdp" event={"ID":"01ab7be9-13bf-4798-bcb0-44ca9923ce0c","Type":"ContainerStarted","Data":"f86a46f6ee2a7f9ee17d8dda4ab7111c497e3fba4259b23f0f886c104d12b1d9"}
Nov 03 06:58:38.288838 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:38.288812    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dxw74" event={"ID":"9c46b893-7db9-4a89-a701-12fa631394fa","Type":"ContainerStarted","Data":"315ffb415c99a86a0964c0c58d3b253fb0cc7533f477ab5729266bff70368aa9"}
Nov 03 06:58:38.291301 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:38.291261    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7" event={"ID":"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3","Type":"ContainerStarted","Data":"eaf0871fb38401a555137dce0262aa444c1004e99c896cba7fd6f4f9b9d04987"}
Nov 03 06:58:38.293030 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:38.293004    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2dp6g" event={"ID":"021ef7d9-672a-4953-bb37-3ca4c271c7ca","Type":"ContainerStarted","Data":"5c43b80fb0672c6835e52e2e21912bd6bddd0d5a7cfb6eb4564f3067ce24f2a6"}
Nov 03 06:58:38.414917 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:38.414867    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pkxfm" podStartSLOduration=6.046291372 podStartE2EDuration="11.414853899s" totalImagesPullingTime="5.368562527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:35.397017997 +0000 UTC m=+8.681512259" watchObservedRunningTime="2026-11-03 06:58:38.414853899 +0000 UTC m=+11.699348219"
Nov 03 06:58:38.535632 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:38.535577    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dxw74" podStartSLOduration=2.988411523 podStartE2EDuration="11.535559957s" totalImagesPullingTime="8.547148434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:38.535111288 +0000 UTC m=+11.819605637" watchObservedRunningTime="2026-11-03 06:58:38.535559957 +0000 UTC m=+11.820054210"
Nov 03 06:58:38.656025 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:38.655962    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2dp6g" podStartSLOduration=3.551900002 podStartE2EDuration="11.65594265s" totalImagesPullingTime="8.104042648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:38.655430807 +0000 UTC m=+11.939925067" watchObservedRunningTime="2026-11-03 06:58:38.65594265 +0000 UTC m=+11.940436912"
Nov 03 06:58:39.039976 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:39.039889    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:39.040754 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:39.040734    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:39.204912 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:39.204711    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:39.205063 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:39.204709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:39.205063 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:39.205025    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:39.205166 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:39.204759    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:39.205166 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:39.205099    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:39.205230 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:39.205195    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:39.297489 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:39.297402    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="f86a46f6ee2a7f9ee17d8dda4ab7111c497e3fba4259b23f0f886c104d12b1d9" exitCode=0
Nov 03 06:58:39.297637 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:39.297530    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmqdp" event={"ID":"01ab7be9-13bf-4798-bcb0-44ca9923ce0c","Type":"ContainerDied","Data":"f86a46f6ee2a7f9ee17d8dda4ab7111c497e3fba4259b23f0f886c104d12b1d9"}
Nov 03 06:58:40.301101 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:40.301059    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gqgl4" event={"ID":"5cea0af3-b692-43ed-8c86-df442fd5378c","Type":"ContainerStarted","Data":"a04a27ebcc595e9c7fdbd8e5cbc764afcfa2d7f0e59d58d695ba07847cd6eb2f"}
Nov 03 06:58:40.303501 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:40.303476    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7" event={"ID":"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3","Type":"ContainerStarted","Data":"1e355c6019fdc27dfb195a626f4dbe04962adabe741213293f6e494b444a3e76"}
Nov 03 06:58:40.305183 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:40.305163    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-krd42" event={"ID":"92fece83-25d4-4543-8489-67583d71ceff","Type":"ContainerStarted","Data":"4a5a276ca5725f2f6ba1055dd734b3e2c5d82f0e33cbb2d6da4d6d96d3372533"}
Nov 03 06:58:40.324493 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:40.324470    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:40.324598 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:40.324578    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 06:58:40.325078 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:40.325056    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:58:40.420249 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:40.420147    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gqgl4" podStartSLOduration=2.796956477 podStartE2EDuration="13.420116918s" totalImagesPullingTime="10.623160441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:40.419685719 +0000 UTC m=+13.704179982" watchObservedRunningTime="2026-11-03 06:58:40.420116918 +0000 UTC m=+13.704611178"
Nov 03 06:58:40.539454 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:40.539407    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7" podStartSLOduration=3.976126935 podStartE2EDuration="13.539392991s" totalImagesPullingTime="9.563266056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:40.539340511 +0000 UTC m=+13.823834771" watchObservedRunningTime="2026-11-03 06:58:40.539392991 +0000 UTC m=+13.823887250"
Nov 03 06:58:40.659548 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:40.659494    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-krd42" podStartSLOduration=5.228774493 podStartE2EDuration="13.659476862s" totalImagesPullingTime="8.430702369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:40.65932749 +0000 UTC m=+13.943821750" watchObservedRunningTime="2026-11-03 06:58:40.659476862 +0000 UTC m=+13.943971129"
Nov 03 06:58:41.204699 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:41.204669    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:41.204869 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:41.204721    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:41.204869 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:41.204676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:41.204869 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:41.204787    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:41.204869 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:41.204852    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:41.205091 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:41.204932    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:41.310147 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:41.310108    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="9e8eb59023191f53ae9a10cab458908ec93a9f411bddc8919df4ef06cf86e0d6" exitCode=0
Nov 03 06:58:41.310658 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:41.310170    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmqdp" event={"ID":"01ab7be9-13bf-4798-bcb0-44ca9923ce0c","Type":"ContainerDied","Data":"9e8eb59023191f53ae9a10cab458908ec93a9f411bddc8919df4ef06cf86e0d6"}
Nov 03 06:58:43.201802 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:43.201552    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:43.202328 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:43.201567    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:43.202328 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:43.201916    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:43.202328 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:43.201630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:43.202328 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:43.201982    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:43.202328 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:43.202062    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:43.317840 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:43.317801    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="a15eb34ea67d369ceb609d372f03ab59e73ec2d37961eca1e677df59a04d1232" exitCode=0
Nov 03 06:58:43.318011 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:43.317851    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmqdp" event={"ID":"01ab7be9-13bf-4798-bcb0-44ca9923ce0c","Type":"ContainerDied","Data":"a15eb34ea67d369ceb609d372f03ab59e73ec2d37961eca1e677df59a04d1232"}
Nov 03 06:58:44.923446 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:44.923299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:44.923446 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:44.923364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:44.924237 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:44.923464    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:44.924237 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:44.923464    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:44.924237 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:44.923521    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret podName:cf458915-e617-486b-8b9f-9019d09ca398 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:00.923504871 +0000 UTC m=+34.207999111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret") pod "global-pull-secret-syncer-pw94d" (UID: "cf458915-e617-486b-8b9f-9019d09ca398") : object "kube-system"/"original-pull-secret" not registered
Nov 03 06:58:44.924237 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:44.923543    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs podName:7d04cba6-64ad-4a4e-8045-3c91d3237a14 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:00.923531361 +0000 UTC m=+34.208025599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs") pod "network-metrics-daemon-mq4wx" (UID: "7d04cba6-64ad-4a4e-8045-3c91d3237a14") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 06:58:45.124845 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:45.124582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5mj\" (UniqueName: \"kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj\") pod \"network-check-target-6bhzs\" (UID: \"af5953b4-e51d-433b-9860-6b077061f77c\") " pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:45.124845 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:45.124760    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 06:58:45.125028 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:45.124870    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 06:58:45.125028 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:45.124884    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2d5mj for pod openshift-network-diagnostics/network-check-target-6bhzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:45.125028 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:45.124942    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj podName:af5953b4-e51d-433b-9860-6b077061f77c nodeName:}" failed. No retries permitted until 2026-11-03 06:59:01.124923808 +0000 UTC m=+34.409418046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2d5mj" (UniqueName: "kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj") pod "network-check-target-6bhzs" (UID: "af5953b4-e51d-433b-9860-6b077061f77c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 06:58:45.201651 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:45.201620    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:45.201651 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:45.201633    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:45.201865 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:45.201756    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:45.201865 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:45.201819    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:45.201981 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:45.201893    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:45.201981 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:45.201970    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:45.325059 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:45.325037    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk" event={"ID":"7bcf499f-e3d7-4146-89f8-74e2ae1685e3","Type":"ContainerStarted","Data":"ac15d82d41eafc9db5fe0cfa795677506232ae3e0c907636e1c368ec78bf6ad2"}
Nov 03 06:58:45.325159 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:45.325100    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk" event={"ID":"7bcf499f-e3d7-4146-89f8-74e2ae1685e3","Type":"ContainerStarted","Data":"ac8e2c885994da94e743ada7d68b4df6d5492e852b6d1bc9a395c924b60e2dac"}
Nov 03 06:58:45.325159 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:45.325150    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk" event={"ID":"7bcf499f-e3d7-4146-89f8-74e2ae1685e3","Type":"ContainerStarted","Data":"4185c53694485ab5c7bbecdbbe304c4811dcfefe2099f92a3c70ac4fb85b75c3"}
Nov 03 06:58:45.325240 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:45.325180    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk" event={"ID":"7bcf499f-e3d7-4146-89f8-74e2ae1685e3","Type":"ContainerStarted","Data":"6e4dd2f07f2c185d4012123010c7f7be31afa1fd0e8dc1a2bfa83c9f91dc429e"}
Nov 03 06:58:45.325240 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:45.325206    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk" event={"ID":"7bcf499f-e3d7-4146-89f8-74e2ae1685e3","Type":"ContainerStarted","Data":"f20da03b5a704ad55c1d94c95e4eab9b86b7886b33f0facb2f9570ecc08b65b0"}
Nov 03 06:58:45.325240 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:45.325232    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk" event={"ID":"7bcf499f-e3d7-4146-89f8-74e2ae1685e3","Type":"ContainerStarted","Data":"3fb1e75ec11b5762e36c1b8cb610903350ed3f3ecb37c8d51594379400f38994"}
Nov 03 06:58:46.329841 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:46.329805    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="27532d715251c5d1588e34c3fb14bb004975a99d4e2833a1cef166a88db2412f" exitCode=0
Nov 03 06:58:46.330215 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:46.329854    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmqdp" event={"ID":"01ab7be9-13bf-4798-bcb0-44ca9923ce0c","Type":"ContainerDied","Data":"27532d715251c5d1588e34c3fb14bb004975a99d4e2833a1cef166a88db2412f"}
Nov 03 06:58:47.202006 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:47.201968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:47.202168 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:47.202056    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:47.202168 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:47.202103    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:47.202168 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:47.202096    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:47.202285 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:47.202219    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:47.202316 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:47.202299    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:47.334884 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:47.334856    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk" event={"ID":"7bcf499f-e3d7-4146-89f8-74e2ae1685e3","Type":"ContainerStarted","Data":"af8828eed3c2440504dd5bbe43e23c8b13a24fa208d7303c66a42d58566cdca1"}
Nov 03 06:58:47.337814 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:47.337790    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="daba2f72354958cd63dfcb2da3f2a0e8e3a4b88fbd5def0ee63f81b98480cc43" exitCode=0
Nov 03 06:58:47.337923 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:47.337830    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmqdp" event={"ID":"01ab7be9-13bf-4798-bcb0-44ca9923ce0c","Type":"ContainerDied","Data":"daba2f72354958cd63dfcb2da3f2a0e8e3a4b88fbd5def0ee63f81b98480cc43"}
Nov 03 06:58:48.342661 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:48.342630    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmqdp" event={"ID":"01ab7be9-13bf-4798-bcb0-44ca9923ce0c","Type":"ContainerStarted","Data":"73ce80e9640874121f829f4e2823177787f90c270f156a560d0577b422af8f5b"}
Nov 03 06:58:48.465580 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:48.465531    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xmqdp" podStartSLOduration=8.873423046 podStartE2EDuration="21.46551879s" totalImagesPullingTime="12.592095744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:48.465339851 +0000 UTC m=+21.749834111" watchObservedRunningTime="2026-11-03 06:58:48.46551879 +0000 UTC m=+21.750013049"
Nov 03 06:58:49.201463 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:49.201421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:49.201674 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:49.201479    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:49.201674 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:49.201578    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:49.201762 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:49.201666    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:49.201762 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:49.201745    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:49.201836 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:49.201818    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:49.348819 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:49.348781    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk" event={"ID":"7bcf499f-e3d7-4146-89f8-74e2ae1685e3","Type":"ContainerStarted","Data":"4459a6ab7b54ef6644d456267341a64ebd370777ac9b4afb9004612ab55cbdff"}
Nov 03 06:58:49.527878 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:49.527824    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk" podStartSLOduration=7.496333301 podStartE2EDuration="22.527811924s" totalImagesPullingTime="15.031478623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:49.527312066 +0000 UTC m=+22.811806325" watchObservedRunningTime="2026-11-03 06:58:49.527811924 +0000 UTC m=+22.812306183"
Nov 03 06:58:50.351955 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:50.351925    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:50.352425 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:50.352002    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:50.352425 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:50.352048    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:50.368963 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:50.368767    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:50.369191 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:50.369175    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:58:51.200885 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:51.200855    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:51.201038 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:51.200859    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:51.201038 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:51.200955    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:51.201038 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:51.200978    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:51.201180 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:51.201062    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:51.201180 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:51.201165    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:51.775295 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:51.775261    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pw94d"]
Nov 03 06:58:51.775735 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:51.775386    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:51.775735 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:51.775501    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:51.781333 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:51.781305    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6bhzs"]
Nov 03 06:58:51.781465 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:51.781417    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:51.781531 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:51.781503    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:51.782076 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:51.782047    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mq4wx"]
Nov 03 06:58:51.782190 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:51.782176    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:51.782304 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:51.782282    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:53.200824 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:53.200794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:53.201294 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:53.200794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:53.201294 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:53.200933    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:53.201294 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:53.200972    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:54.201561 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:54.201528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:54.202002 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:54.201642    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:55.201491 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:55.201459    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:55.201675 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:55.201523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:55.205329 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:55.205288    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:55.205457 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:55.205405    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:56.200957 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:56.200915    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:56.201117 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:56.201046    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw94d" podUID="cf458915-e617-486b-8b9f-9019d09ca398"
Nov 03 06:58:57.202005 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.201977    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:57.202446 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:57.202064    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mq4wx" podUID="7d04cba6-64ad-4a4e-8045-3c91d3237a14"
Nov 03 06:58:57.202446 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.202186    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:57.202446 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:57.202309    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bhzs" podUID="af5953b4-e51d-433b-9860-6b077061f77c"
Nov 03 06:58:57.474268 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.474237    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeReady"
Nov 03 06:58:57.474398 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.474371    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 06:58:57.589108 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.589078    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pcxdc"]
Nov 03 06:58:57.592084 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.592066    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.638625 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.638600    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6vxzv"]
Nov 03 06:58:57.641627 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.641611    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vprr7"]
Nov 03 06:58:57.641758 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.641729    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:58:57.645225 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.645205    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcxdc"]
Nov 03 06:58:57.645310 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.645236    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6vxzv"]
Nov 03 06:58:57.645346 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.645312    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.651121 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.651056    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 06:58:57.651388 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.651371    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-l2dln-4d54l"]
Nov 03 06:58:57.651476 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.651458    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 06:58:57.651616 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.651598    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nd4b5\""
Nov 03 06:58:57.654429 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.654412    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.688399 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.688372    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vprr7"]
Nov 03 06:58:57.710959 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.710939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.711008 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.710982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f4ea98-f511-4031-8b8b-7d057e1433cd-tmp-dir\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.711045 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.711021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f4ea98-f511-4031-8b8b-7d057e1433cd-config-volume\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.711113 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.711101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrkvw\" (UniqueName: \"kubernetes.io/projected/a5f4ea98-f511-4031-8b8b-7d057e1433cd-kube-api-access-jrkvw\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.743579 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.743556    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-l2dln-4d54l"]
Nov 03 06:58:57.750944 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.750924    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 06:58:57.750944 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.750939    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 06:58:57.751107 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.750966    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 06:58:57.751107 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.751040    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 06:58:57.751239 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.751152    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 06:58:57.751239 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.751187    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 06:58:57.751239 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.751211    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 06:58:57.751324 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.751271    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 06:58:57.751402 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.751387    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7dznd\""
Nov 03 06:58:57.751437 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.751403    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8gp29\""
Nov 03 06:58:57.751480 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.751444    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kl8t2\""
Nov 03 06:58:57.801624 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.801602    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 06:58:57.811435 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.811487 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab43a143-464a-4155-98e7-3f7c480aeca6-data-volume\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.811487 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v754j\" (UniqueName: \"kubernetes.io/projected/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-kube-api-access-v754j\") pod \"ingress-canary-6vxzv\" (UID: \"2cee886c-6e9d-431b-9cda-0f8cd618cf2b\") " pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:58:57.811551 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811517    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f4ea98-f511-4031-8b8b-7d057e1433cd-config-volume\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.811588 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811546    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.811673 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrkvw\" (UniqueName: \"kubernetes.io/projected/a5f4ea98-f511-4031-8b8b-7d057e1433cd-kube-api-access-jrkvw\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.811703 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-service-ca-bundle\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.811734 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab43a143-464a-4155-98e7-3f7c480aeca6-crio-socket\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.811734 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b57d\" (UniqueName: \"kubernetes.io/projected/3c56f558-2833-421b-a2a4-fd023736ea7c-kube-api-access-8b57d\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.811852 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c56f558-2833-421b-a2a4-fd023736ea7c-archives-path\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.811926 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.811963 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-trusted-ca-bundle\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.812009 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.811991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-cert\") pod \"ingress-canary-6vxzv\" (UID: \"2cee886c-6e9d-431b-9cda-0f8cd618cf2b\") " pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:58:57.812042 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:57.812009    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 06:58:57.812069 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:57.812051    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls podName:a5f4ea98-f511-4031-8b8b-7d057e1433cd nodeName:}" failed. No retries permitted until 2026-11-03 06:58:58.312038677 +0000 UTC m=+31.596532944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls") pod "dns-default-pcxdc" (UID: "a5f4ea98-f511-4031-8b8b-7d057e1433cd") : secret "dns-default-metrics-tls" not found
Nov 03 06:58:57.812104 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.812067    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f4ea98-f511-4031-8b8b-7d057e1433cd-config-volume\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.812173 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.812161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b44wp\" (UniqueName: \"kubernetes.io/projected/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-api-access-b44wp\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.812209 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.812189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f4ea98-f511-4031-8b8b-7d057e1433cd-tmp-dir\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.812466 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.812448    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f4ea98-f511-4031-8b8b-7d057e1433cd-tmp-dir\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.876061 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.876027    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrkvw\" (UniqueName: \"kubernetes.io/projected/a5f4ea98-f511-4031-8b8b-7d057e1433cd-kube-api-access-jrkvw\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:57.913170 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v754j\" (UniqueName: \"kubernetes.io/projected/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-kube-api-access-v754j\") pod \"ingress-canary-6vxzv\" (UID: \"2cee886c-6e9d-431b-9cda-0f8cd618cf2b\") " pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:58:57.913170 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.913375 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-service-ca-bundle\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.913375 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab43a143-464a-4155-98e7-3f7c480aeca6-crio-socket\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.913375 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913235    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b57d\" (UniqueName: \"kubernetes.io/projected/3c56f558-2833-421b-a2a4-fd023736ea7c-kube-api-access-8b57d\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.913375 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c56f558-2833-421b-a2a4-fd023736ea7c-archives-path\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.913375 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:57.913305    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 06:58:57.913620 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:57.913389    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:58.413366789 +0000 UTC m=+31.697861056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : configmap "kube-rbac-proxy" not found
Nov 03 06:58:57.913620 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-trusted-ca-bundle\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.913620 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913427    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab43a143-464a-4155-98e7-3f7c480aeca6-crio-socket\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.913620 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-cert\") pod \"ingress-canary-6vxzv\" (UID: \"2cee886c-6e9d-431b-9cda-0f8cd618cf2b\") " pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:58:57.913620 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b44wp\" (UniqueName: \"kubernetes.io/projected/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-api-access-b44wp\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.913866 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.913866 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab43a143-464a-4155-98e7-3f7c480aeca6-data-volume\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.913866 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:57.913756    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 06:58:57.913866 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:57.913815    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:58.413796874 +0000 UTC m=+31.698291135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : secret "insights-runtime-extractor-tls" not found
Nov 03 06:58:57.914046 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913890    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-service-ca-bundle\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.914046 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.913975    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab43a143-464a-4155-98e7-3f7c480aeca6-data-volume\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.914046 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.914039    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c56f558-2833-421b-a2a4-fd023736ea7c-archives-path\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.914641 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.914624    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-trusted-ca-bundle\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:57.915919 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.915902    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-cert\") pod \"ingress-canary-6vxzv\" (UID: \"2cee886c-6e9d-431b-9cda-0f8cd618cf2b\") " pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:58:57.976845 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.976810    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b44wp\" (UniqueName: \"kubernetes.io/projected/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-api-access-b44wp\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:57.976997 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.976941    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v754j\" (UniqueName: \"kubernetes.io/projected/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-kube-api-access-v754j\") pod \"ingress-canary-6vxzv\" (UID: \"2cee886c-6e9d-431b-9cda-0f8cd618cf2b\") " pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:58:57.976997 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:57.976980    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b57d\" (UniqueName: \"kubernetes.io/projected/3c56f558-2833-421b-a2a4-fd023736ea7c-kube-api-access-8b57d\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:58.096064 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.096032    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6484x"]
Nov 03 06:58:58.100561 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.100540    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:58:58.155151 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.155100    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 06:58:58.155151 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.155118    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 06:58:58.155151 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.155103    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 06:58:58.155344 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.155278    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 06:58:58.155344 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.155285    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cb66z\""
Nov 03 06:58:58.158920 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.158891    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6484x"]
Nov 03 06:58:58.201552 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.201483    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:58:58.216315 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.216274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-key\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:58:58.216813 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.216349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-cabundle\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:58:58.216813 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.216437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfk5v\" (UniqueName: \"kubernetes.io/projected/03645044-2dfb-4a33-8a26-c3c5684cbe09-kube-api-access-bfk5v\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:58:58.256922 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.256906    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kx4sk\""
Nov 03 06:58:58.257022 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.256977    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 06:58:58.259057 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.259044    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:58:58.272153 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.272122    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:58:58.317067 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.316932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-cabundle\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:58:58.317067 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.316995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfk5v\" (UniqueName: \"kubernetes.io/projected/03645044-2dfb-4a33-8a26-c3c5684cbe09-kube-api-access-bfk5v\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:58:58.317067 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.317040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:58.317364 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.317074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-key\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:58:58.317426 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:58.317381    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 06:58:58.317478 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:58.317453    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls podName:a5f4ea98-f511-4031-8b8b-7d057e1433cd nodeName:}" failed. No retries permitted until 2026-11-03 06:58:59.317433362 +0000 UTC m=+32.601927600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls") pod "dns-default-pcxdc" (UID: "a5f4ea98-f511-4031-8b8b-7d057e1433cd") : secret "dns-default-metrics-tls" not found
Nov 03 06:58:58.318072 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.317975    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-cabundle\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:58:58.320490 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.320439    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-key\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:58:58.379624 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.379596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfk5v\" (UniqueName: \"kubernetes.io/projected/03645044-2dfb-4a33-8a26-c3c5684cbe09-kube-api-access-bfk5v\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:58:58.411449 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.411431    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:58:58.418241 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.418218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:58.418333 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.418253    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:58.418333 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:58.418317    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 06:58:58.418447 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:58.418352    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 06:58:58.418447 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:58.418364    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:59.418351814 +0000 UTC m=+32.702846052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : configmap "kube-rbac-proxy" not found
Nov 03 06:58:58.418447 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:58.418398    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:59.418382111 +0000 UTC m=+32.702876366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : secret "insights-runtime-extractor-tls" not found
Nov 03 06:58:58.647033 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.646994    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-l2dln-4d54l"]
Nov 03 06:58:58.651996 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:58.651965    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c56f558_2833_421b_a2a4_fd023736ea7c.slice/crio-3c147a484bb1e1327c07a9e5d323904fbcc70a215e17ba690b46198eaf8855b0 WatchSource:0}: Error finding container 3c147a484bb1e1327c07a9e5d323904fbcc70a215e17ba690b46198eaf8855b0: Status 404 returned error can't find the container with id 3c147a484bb1e1327c07a9e5d323904fbcc70a215e17ba690b46198eaf8855b0
Nov 03 06:58:58.692946 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.692923    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6vxzv"]
Nov 03 06:58:58.695638 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:58.695613    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cee886c_6e9d_431b_9cda_0f8cd618cf2b.slice/crio-688b3cd0e94a268b40abd853ace4742279560e36d41986e06efb1656d7aed4eb WatchSource:0}: Error finding container 688b3cd0e94a268b40abd853ace4742279560e36d41986e06efb1656d7aed4eb: Status 404 returned error can't find the container with id 688b3cd0e94a268b40abd853ace4742279560e36d41986e06efb1656d7aed4eb
Nov 03 06:58:58.777915 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:58.777853    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6484x"]
Nov 03 06:58:58.780462 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:58:58.780439    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03645044_2dfb_4a33_8a26_c3c5684cbe09.slice/crio-add3c286ab668399a7118f1f8d30168cffcd5d439f8abba785e4a016a4b0c4d4 WatchSource:0}: Error finding container add3c286ab668399a7118f1f8d30168cffcd5d439f8abba785e4a016a4b0c4d4: Status 404 returned error can't find the container with id add3c286ab668399a7118f1f8d30168cffcd5d439f8abba785e4a016a4b0c4d4
Nov 03 06:58:59.201816 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.201789    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:58:59.201816 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.201813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:58:59.256362 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.256342    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 06:58:59.256678 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.256459    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dftjg\""
Nov 03 06:58:59.256678 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.256508    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tq2gc\""
Nov 03 06:58:59.256678 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.256613    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 06:58:59.256678 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.256662    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 06:58:59.325412 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.325391    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:58:59.325535 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:59.325522    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 06:58:59.325590 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:59.325580    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls podName:a5f4ea98-f511-4031-8b8b-7d057e1433cd nodeName:}" failed. No retries permitted until 2026-11-03 06:59:01.325566132 +0000 UTC m=+34.610060384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls") pod "dns-default-pcxdc" (UID: "a5f4ea98-f511-4031-8b8b-7d057e1433cd") : secret "dns-default-metrics-tls" not found
Nov 03 06:58:59.373322 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.373296    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6484x" event={"ID":"03645044-2dfb-4a33-8a26-c3c5684cbe09","Type":"ContainerStarted","Data":"add3c286ab668399a7118f1f8d30168cffcd5d439f8abba785e4a016a4b0c4d4"}
Nov 03 06:58:59.374452 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.374434    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l2dln-4d54l" event={"ID":"3c56f558-2833-421b-a2a4-fd023736ea7c","Type":"ContainerStarted","Data":"3c147a484bb1e1327c07a9e5d323904fbcc70a215e17ba690b46198eaf8855b0"}
Nov 03 06:58:59.375489 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.375471    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6vxzv" event={"ID":"2cee886c-6e9d-431b-9cda-0f8cd618cf2b","Type":"ContainerStarted","Data":"688b3cd0e94a268b40abd853ace4742279560e36d41986e06efb1656d7aed4eb"}
Nov 03 06:58:59.426108 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.426084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:59.426193 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:58:59.426180    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:58:59.426234 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:59.426196    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 06:58:59.426273 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:59.426246    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:01.426231639 +0000 UTC m=+34.710725893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : configmap "kube-rbac-proxy" not found
Nov 03 06:58:59.426315 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:59.426272    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 06:58:59.426315 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:58:59.426307    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:01.426297351 +0000 UTC m=+34.710791589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:00.910062 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:00.910029    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9v5rw_b5aebe2f-412e-4d2a-b644-7fd13bb62e07/konnectivity-agent/0.log" limit=1024
Nov 03 06:59:00.939242 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:00.939211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:59:00.939425 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:00.939299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:59:00.939425 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:00.939367    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 06:59:00.939558 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:00.939434    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs podName:7d04cba6-64ad-4a4e-8045-3c91d3237a14 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:32.939416357 +0000 UTC m=+66.223910614 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs") pod "network-metrics-daemon-mq4wx" (UID: "7d04cba6-64ad-4a4e-8045-3c91d3237a14") : secret "metrics-daemon-secret" not found
Nov 03 06:59:00.941918 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:00.941893    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:59:01.141242 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:01.141199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5mj\" (UniqueName: \"kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj\") pod \"network-check-target-6bhzs\" (UID: \"af5953b4-e51d-433b-9860-6b077061f77c\") " pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:59:01.144431 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:01.144408    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5mj\" (UniqueName: \"kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj\") pod \"network-check-target-6bhzs\" (UID: \"af5953b4-e51d-433b-9860-6b077061f77c\") " pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:59:01.168293 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:01.168228    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9v5rw_b5aebe2f-412e-4d2a-b644-7fd13bb62e07/konnectivity-agent/0.log" limit=1024
Nov 03 06:59:01.212565 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:01.212530    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:59:01.316563 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:01.315576    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:59:01.342346 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:01.342317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:01.342471 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:01.342443    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 06:59:01.342531 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:01.342494    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls podName:a5f4ea98-f511-4031-8b8b-7d057e1433cd nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.342479933 +0000 UTC m=+38.626974175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls") pod "dns-default-pcxdc" (UID: "a5f4ea98-f511-4031-8b8b-7d057e1433cd") : secret "dns-default-metrics-tls" not found
Nov 03 06:59:01.443499 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:01.443297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:01.443499 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:01.443478    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:01.443746 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:01.443570    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.443549322 +0000 UTC m=+38.728043562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:01.444077 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:01.443857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:01.444077 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:01.443993    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 06:59:01.444077 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:01.444056    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.444040006 +0000 UTC m=+38.728534251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : configmap "kube-rbac-proxy" not found
Nov 03 06:59:01.627081 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:01.627051    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pw94d"]
Nov 03 06:59:01.666824 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:59:01.666767    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf458915_e617_486b_8b9f_9019d09ca398.slice/crio-c9703783abab36c8ee8998cab1e352c8eea7c4b8c85bc262ab1ade211017da9a WatchSource:0}: Error finding container c9703783abab36c8ee8998cab1e352c8eea7c4b8c85bc262ab1ade211017da9a: Status 404 returned error can't find the container with id c9703783abab36c8ee8998cab1e352c8eea7c4b8c85bc262ab1ade211017da9a
Nov 03 06:59:01.732148 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:01.732056    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6bhzs"]
Nov 03 06:59:01.735451 ip-10-0-129-24 kubenswrapper[2576]: W1103 06:59:01.735422    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5953b4_e51d_433b_9860_6b077061f77c.slice/crio-d0260ec75b016e0862207d2a0c8467c296d6aab04a6ca3bf6560b514f4214a47 WatchSource:0}: Error finding container d0260ec75b016e0862207d2a0c8467c296d6aab04a6ca3bf6560b514f4214a47: Status 404 returned error can't find the container with id d0260ec75b016e0862207d2a0c8467c296d6aab04a6ca3bf6560b514f4214a47
Nov 03 06:59:02.386336 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:02.386300    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6484x" event={"ID":"03645044-2dfb-4a33-8a26-c3c5684cbe09","Type":"ContainerStarted","Data":"a0ec548b0893563d6363804b64e5f7a410655bab5c91d3d728e1c3381d5b2c74"}
Nov 03 06:59:02.387654 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:02.387630    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6bhzs" event={"ID":"af5953b4-e51d-433b-9860-6b077061f77c","Type":"ContainerStarted","Data":"d0260ec75b016e0862207d2a0c8467c296d6aab04a6ca3bf6560b514f4214a47"}
Nov 03 06:59:02.389121 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:02.389095    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l2dln-4d54l" event={"ID":"3c56f558-2833-421b-a2a4-fd023736ea7c","Type":"ContainerStarted","Data":"8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e"}
Nov 03 06:59:02.390160 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:02.390120    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pw94d" event={"ID":"cf458915-e617-486b-8b9f-9019d09ca398","Type":"ContainerStarted","Data":"c9703783abab36c8ee8998cab1e352c8eea7c4b8c85bc262ab1ade211017da9a"}
Nov 03 06:59:02.504173 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:02.504101    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-6484x" podStartSLOduration=1.071523775 podStartE2EDuration="4.504083039s" totalImagesPullingTime="3.432559264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:02.503487518 +0000 UTC m=+35.787981802" watchObservedRunningTime="2026-11-03 06:59:02.504083039 +0000 UTC m=+35.788577301"
Nov 03 06:59:02.621285 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:02.621224    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-l2dln-4d54l" podStartSLOduration=2.072320489 podStartE2EDuration="5.621205867s" totalImagesPullingTime="3.548885378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:02.62076107 +0000 UTC m=+35.905255359" watchObservedRunningTime="2026-11-03 06:59:02.621205867 +0000 UTC m=+35.905700128"
Nov 03 06:59:03.395031 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:03.394986    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6vxzv" event={"ID":"2cee886c-6e9d-431b-9cda-0f8cd618cf2b","Type":"ContainerStarted","Data":"e5ccf4cbc2fbf558d3c6a1fd263f77eb18473d261d078823241aa4edd1173f25"}
Nov 03 06:59:05.390885 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:05.390845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:05.391340 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:05.391027    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 06:59:05.391340 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:05.391103    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls podName:a5f4ea98-f511-4031-8b8b-7d057e1433cd nodeName:}" failed. No retries permitted until 2026-11-03 06:59:13.391082879 +0000 UTC m=+46.675577132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls") pod "dns-default-pcxdc" (UID: "a5f4ea98-f511-4031-8b8b-7d057e1433cd") : secret "dns-default-metrics-tls" not found
Nov 03 06:59:05.491589 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:05.491553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:05.491758 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:05.491613    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:05.491758 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:05.491660    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:05.491758 ip-10-0-129-24 kubenswrapper[2576]: E1103 06:59:05.491732    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:13.49171178 +0000 UTC m=+46.776206033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:05.492179 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:05.492161    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:06.405445 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:06.405403    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6bhzs" event={"ID":"af5953b4-e51d-433b-9860-6b077061f77c","Type":"ContainerStarted","Data":"b45b39f89bc14ab693baeb260d1e428df82142c9f76215653d7ea6bba6e7fba5"}
Nov 03 06:59:06.405876 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:06.405491    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:59:06.522445 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:06.522391    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6vxzv" podStartSLOduration=5.742628307 podStartE2EDuration="9.522372738s" totalImagesPullingTime="3.779744431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:03.519944265 +0000 UTC m=+36.804438537" watchObservedRunningTime="2026-11-03 06:59:06.522372738 +0000 UTC m=+39.806866998"
Nov 03 06:59:06.522683 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:06.522661    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6bhzs" podStartSLOduration=35.355230403 podStartE2EDuration="39.522656481s" totalImagesPullingTime="4.167426078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:06.521570359 +0000 UTC m=+39.806064621" watchObservedRunningTime="2026-11-03 06:59:06.522656481 +0000 UTC m=+39.807150741"
Nov 03 06:59:07.028085 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:07.028055    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dxw74_9c46b893-7db9-4a89-a701-12fa631394fa/dns-node-resolver/0.log"
Nov 03 06:59:07.409545 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:07.409506    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pw94d" event={"ID":"cf458915-e617-486b-8b9f-9019d09ca398","Type":"ContainerStarted","Data":"5e5c431903d2b4a7346c53bb42cef5bfffb900c97cde5efde469a20ececf9c55"}
Nov 03 06:59:07.597349 ip-10-0-129-24 kubenswrapper[2576]: I1103 06:59:07.596884    2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 06:59:07.597047 ip-10-0-129-24 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 06:59:07.609566 ip-10-0-129-24 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 06:59:07.609730 ip-10-0-129-24 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 06:59:07.609779 ip-10-0-129-24 systemd[1]: kubelet.service: Consumed 2.506s CPU time.
Nov 03 06:59:07.623777 ip-10-0-129-24 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 06:59:07.725588 ip-10-0-129-24 kubenswrapper[4470]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 06:59:07.725588 ip-10-0-129-24 kubenswrapper[4470]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 06:59:07.725588 ip-10-0-129-24 kubenswrapper[4470]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 06:59:07.725588 ip-10-0-129-24 kubenswrapper[4470]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 06:59:07.727700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727619    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:59:07.727700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727700    4470 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727705    4470 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727708    4470 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727711    4470 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727714    4470 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727717    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727719    4470 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727722    4470 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727724    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727727    4470 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727730    4470 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727732    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727735    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727738    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727741    4470 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727743    4470 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727746    4470 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727749    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727752    4470 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:59:07.727775 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727754    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727757    4470 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727759    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727762    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727765    4470 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727767    4470 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727770    4470 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727773    4470 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727775    4470 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727778    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727780    4470 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727783    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727786    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727789    4470 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727791    4470 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727794    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727796    4470 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727799    4470 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727802    4470 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727804    4470 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:59:07.728284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727807    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727809    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727812    4470 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727814    4470 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727817    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727819    4470 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727821    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727824    4470 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727826    4470 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727829    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727832    4470 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727835    4470 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727837    4470 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727840    4470 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727842    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727845    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727848    4470 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727850    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727853    4470 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:59:07.728908 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727856    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727858    4470 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727861    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727863    4470 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727866    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727868    4470 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727871    4470 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727873    4470 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727876    4470 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727879    4470 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727883    4470 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727886    4470 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727889    4470 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727892    4470 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727894    4470 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727897    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727900    4470 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727903    4470 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727906    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:59:07.729687 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727909    4470 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727911    4470 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727913    4470 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727916    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727918    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727922    4470 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727924    4470 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727927    4470 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727930    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727932    4470 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727935    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727937    4470 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727940    4470 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727944    4470 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727948    4470 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727951    4470 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727954    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727957    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727960    4470 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727963    4470 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:59:07.730462 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727965    4470 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727968    4470 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727970    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727973    4470 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727975    4470 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727978    4470 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727980    4470 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.727983    4470 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728809    4470 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728819    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728823    4470 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728826    4470 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728829    4470 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728832    4470 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728834    4470 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728837    4470 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728839    4470 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728842    4470 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728844    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:59:07.731274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728849    4470 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728853    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728857    4470 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728860    4470 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728863    4470 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728866    4470 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728868    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728871    4470 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728874    4470 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728877    4470 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728880    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728885    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728889    4470 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728894    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728898    4470 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728900    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728903    4470 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728905    4470 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728908    4470 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:59:07.732028 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728910    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728915    4470 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728923    4470 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728926    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728929    4470 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728932    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728935    4470 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728938    4470 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728940    4470 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728943    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728945    4470 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728948    4470 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728950    4470 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728954    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728956    4470 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728960    4470 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.728965    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729021    4470 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729025    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729027    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:59:07.732803 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729031    4470 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729033    4470 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729037    4470 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729041    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729045    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729049    4470 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729054    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729057    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729059    4470 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729062    4470 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729064    4470 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729067    4470 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729070    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729073    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729077    4470 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729080    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729082    4470 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729085    4470 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729088    4470 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729091    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:59:07.733770 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729094    4470 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729096    4470 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729099    4470 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729101    4470 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729104    4470 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729106    4470 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729109    4470 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729111    4470 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729114    4470 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729117    4470 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729121    4470 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729126    4470 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729144    4470 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729149    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729152    4470 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729155    4470 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729158    4470 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729160    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729162    4470 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729165    4470 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:59:07.734593 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729167    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729170    4470 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729172    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729175    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729177    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729180    4470 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729184    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729187    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729190    4470 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729192    4470 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729195    4470 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729201    4470 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729206    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729212    4470 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729215    4470 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729218    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729766    4470 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729772    4470 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729775    4470 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729778    4470 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:59:07.735415 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729781    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729784    4470 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729786    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729789    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729791    4470 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729794    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729796    4470 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729799    4470 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729801    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729804    4470 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729806    4470 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729809    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729811    4470 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729813    4470 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729816    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729818    4470 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729822    4470 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729827    4470 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729832    4470 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729837    4470 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:59:07.736284 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729840    4470 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729842    4470 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729844    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729848    4470 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729851    4470 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729853    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729855    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729858    4470 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729861    4470 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729863    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729865    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729868    4470 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729870    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729872    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729875    4470 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729877    4470 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729880    4470 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729882    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729884    4470 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:59:07.737089 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729887    4470 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729889    4470 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729891    4470 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729894    4470 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729896    4470 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729899    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729904    4470 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729909    4470 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729913    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729917    4470 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729922    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729926    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729929    4470 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729931    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729934    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729937    4470 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729940    4470 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729947    4470 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729949    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:59:07.737918 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729952    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729954    4470 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729957    4470 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729959    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729962    4470 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729964    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729967    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729969    4470 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729972    4470 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729975    4470 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729980    4470 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729986    4470 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729993    4470 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.729996    4470 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730000    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730002    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730005    4470 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730008    4470 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730010    4470 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:59:07.738699 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730013    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730015    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730018    4470 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730020    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730024    4470 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730028    4470 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730031    4470 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730034    4470 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730037    4470 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730039    4470 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730042    4470 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730045    4470 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730048    4470 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730051    4470 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730053    4470 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730057    4470 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730061    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730065    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730069    4470 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730071    4470 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:59:07.739465 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730074    4470 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730078    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730080    4470 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730082    4470 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730085    4470 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730210    4470 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730218    4470 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730225    4470 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730230    4470 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730238    4470 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730244    4470 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730251    4470 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730257    4470 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730261    4470 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730264    4470 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730268    4470 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730271    4470 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730274    4470 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730277    4470 flags.go:64] FLAG: --cgroup-root=""
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730279    4470 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730282    4470 flags.go:64] FLAG: --client-ca-file=""
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730285    4470 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730288    4470 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730292    4470 flags.go:64] FLAG: --cluster-domain=""
Nov 03 06:59:07.740301 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730295    4470 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730298    4470 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730301    4470 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730305    4470 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730308    4470 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730311    4470 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730314    4470 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730319    4470 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730324    4470 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730329    4470 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730334    4470 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730341    4470 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730345    4470 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730349    4470 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730352    4470 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730355    4470 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730358    4470 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730361    4470 flags.go:64] FLAG: --enable-server="true"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730364    4470 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730369    4470 flags.go:64] FLAG: --event-burst="100"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730372    4470 flags.go:64] FLAG: --event-qps="50"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730375    4470 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730378    4470 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730380    4470 flags.go:64] FLAG: --eviction-hard=""
Nov 03 06:59:07.741249 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730385    4470 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730387    4470 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730390    4470 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730393    4470 flags.go:64] FLAG: --eviction-soft=""
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730396    4470 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730400    4470 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730405    4470 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730410    4470 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730415    4470 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730420    4470 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730423    4470 flags.go:64] FLAG: --feature-gates=""
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730431    4470 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730434    4470 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730437    4470 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730440    4470 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730443    4470 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730446    4470 flags.go:64] FLAG: --help="false"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730449    4470 flags.go:64] FLAG: --hostname-override="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730452    4470 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730458    4470 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730462    4470 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730466    4470 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730469    4470 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 06:59:07.742242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730472    4470 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730475    4470 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730479    4470 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730483    4470 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730488    4470 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730494    4470 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730498    4470 flags.go:64] FLAG: --kube-reserved=""
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730503    4470 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730506    4470 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730509    4470 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730512    4470 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730514    4470 flags.go:64] FLAG: --lock-file=""
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730517    4470 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730520    4470 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730523    4470 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730529    4470 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730532    4470 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730535    4470 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730538    4470 flags.go:64] FLAG: --logging-format="text"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730541    4470 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730544    4470 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730547    4470 flags.go:64] FLAG: --manifest-url=""
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730550    4470 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730554    4470 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730557    4470 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 06:59:07.743334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730561    4470 flags.go:64] FLAG: --max-pods="110"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730565    4470 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730570    4470 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730577    4470 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730584    4470 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730588    4470 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730591    4470 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730594    4470 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730601    4470 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730604    4470 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730606    4470 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730609    4470 flags.go:64] FLAG: --pod-cidr=""
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730612    4470 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730615    4470 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730618    4470 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730621    4470 flags.go:64] FLAG: --port="10250"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730624    4470 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730627    4470 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-021bc18b8e84ebaf3"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730630    4470 flags.go:64] FLAG: --qos-reserved=""
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730633    4470 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730636    4470 flags.go:64] FLAG: --register-node="true"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730638    4470 flags.go:64] FLAG: --register-with-taints=""
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730642    4470 flags.go:64] FLAG: --registry-burst="10"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730646    4470 flags.go:64] FLAG: --registry-qps="5"
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730651    4470 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730656    4470 flags.go:64] FLAG: --reserved-memory=""
Nov 03 06:59:07.744325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730662    4470 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730667    4470 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730671    4470 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730674    4470 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730677    4470 flags.go:64] FLAG: --runonce="false"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730697    4470 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730701    4470 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730705    4470 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730708    4470 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730712    4470 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730717    4470 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730720    4470 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730723    4470 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730726    4470 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730731    4470 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730736    4470 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730741    4470 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730746    4470 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730750    4470 flags.go:64] FLAG: --system-cgroups=""
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730753    4470 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730758    4470 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730761    4470 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730764    4470 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730767    4470 flags.go:64] FLAG: --tls-min-version=""
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730770    4470 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 06:59:07.745361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730773    4470 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730776    4470 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730778    4470 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730781    4470 flags.go:64] FLAG: --v="2"
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730786    4470 flags.go:64] FLAG: --version="false"
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730789    4470 flags.go:64] FLAG: --vmodule=""
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730794    4470 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.730797    4470 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730951    4470 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730956    4470 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730960    4470 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730963    4470 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730966    4470 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730969    4470 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730971    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730975    4470 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730982    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730988    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730993    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.730997    4470 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:59:07.746417 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731001    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731005    4470 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731008    4470 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731011    4470 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731014    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731018    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731020    4470 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731023    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731027    4470 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731030    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731033    4470 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731036    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731039    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731041    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731044    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731046    4470 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731049    4470 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731051    4470 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731053    4470 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731057    4470 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:59:07.747303 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731061    4470 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731066    4470 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731071    4470 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731076    4470 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731079    4470 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731081    4470 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731084    4470 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731086    4470 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731090    4470 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731094    4470 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731097    4470 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731100    4470 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731102    4470 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731105    4470 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731107    4470 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731110    4470 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731112    4470 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731115    4470 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731117    4470 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731120    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:59:07.748128 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731122    4470 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731125    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731127    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731147    4470 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731152    4470 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731156    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731160    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731162    4470 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731165    4470 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731167    4470 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731170    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731172    4470 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731175    4470 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731177    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731180    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731182    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731185    4470 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731188    4470 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731190    4470 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731193    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:59:07.748942 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731197    4470 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731201    4470 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731203    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731206    4470 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731208    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731211    4470 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731214    4470 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731216    4470 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731219    4470 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731222    4470 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731227    4470 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731231    4470 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731236    4470 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731240    4470 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731243    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731246    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731248    4470 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731251    4470 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731253    4470 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731256    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:59:07.749753 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731258    4470 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731261    4470 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731263    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731265    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731268    4470 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731270    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731273    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731289    4470 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731294    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731297    4470 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731300    4470 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731303    4470 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731310    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731316    4470 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.731348    4470 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731385    4470 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:59:07.750558 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731391    4470 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731397    4470 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731403    4470 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731408    4470 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731410    4470 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731413    4470 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731416    4470 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731419    4470 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731421    4470 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731424    4470 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731426    4470 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731428    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731431    4470 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731434    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731436    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731439    4470 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731441    4470 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731443    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731446    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731448    4470 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:59:07.751232 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731451    4470 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731453    4470 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731456    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731458    4470 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731461    4470 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731463    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731466    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731469    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731475    4470 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731480    4470 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731486    4470 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731491    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731494    4470 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731496    4470 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731499    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731501    4470 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731504    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731506    4470 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731509    4470 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731511    4470 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:59:07.752066 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731514    4470 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731516    4470 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731519    4470 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731521    4470 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731524    4470 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731526    4470 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731528    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731531    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731533    4470 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731535    4470 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731544    4470 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731548    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731551    4470 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731555    4470 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731559    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731563    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731567    4470 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731572    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731575    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:59:07.752895 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731578    4470 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731582    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731585    4470 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731589    4470 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731592    4470 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731594    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731597    4470 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731599    4470 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731603    4470 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731605    4470 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731607    4470 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731610    4470 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731612    4470 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731615    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731617    4470 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731620    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731622    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731625    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731629    4470 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:59:07.753667 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731632    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731635    4470 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731639    4470 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731644    4470 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731648    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731653    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731657    4470 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731660    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731663    4470 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731667    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731670    4470 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731672    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731675    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731677    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731683    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731686    4470 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731689    4470 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731692    4470 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731694    4470 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731697    4470 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:59:07.754468 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731699    4470 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731702    4470 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731705    4470 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731707    4470 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731709    4470 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731712    4470 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.731714    4470 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.735588    4470 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.735602    4470 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.735620    4470 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.735625    4470 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735680    4470 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735685    4470 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735688    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735691    4470 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735694    4470 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735697    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735699    4470 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735702    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:59:07.755280 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735704    4470 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735707    4470 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735709    4470 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735712    4470 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735714    4470 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735717    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735720    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735723    4470 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735726    4470 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735729    4470 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735731    4470 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735734    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735736    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735739    4470 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735742    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735745    4470 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735748    4470 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735752    4470 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735756    4470 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:59:07.756046 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735760    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735763    4470 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735766    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735768    4470 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735771    4470 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735773    4470 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735776    4470 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735779    4470 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735781    4470 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735784    4470 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735786    4470 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735789    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735791    4470 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735793    4470 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735796    4470 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735798    4470 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735801    4470 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735803    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735806    4470 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:59:07.756939 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735808    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735811    4470 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735814    4470 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735816    4470 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735819    4470 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735822    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735824    4470 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735826    4470 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735829    4470 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735832    4470 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735835    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735837    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735840    4470 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735842    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735845    4470 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735847    4470 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735850    4470 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735852    4470 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735855    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:59:07.757700 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735857    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735860    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735862    4470 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735865    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735867    4470 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735869    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735872    4470 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735874    4470 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735877    4470 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735879    4470 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735881    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735885    4470 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735887    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735890    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735892    4470 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735894    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735897    4470 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735900    4470 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735903    4470 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735905    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:59:07.758547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735909    4470 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735913    4470 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735916    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735919    4470 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735921    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735924    4470 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735926    4470 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735928    4470 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735931    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735933    4470 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735936    4470 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735938    4470 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735941    4470 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735943    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735946    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735948    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735951    4470 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735953    4470 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735956    4470 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735958    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:59:07.759373 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.735961    4470 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.735987    4470 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736122    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736127    4470 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736147    4470 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736151    4470 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736154    4470 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736156    4470 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736159    4470 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736162    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736165    4470 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736168    4470 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736171    4470 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736173    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736176    4470 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736178    4470 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:59:07.760176 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736181    4470 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736183    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736186    4470 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736188    4470 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736191    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736193    4470 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736195    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736198    4470 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736200    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736203    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736206    4470 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736208    4470 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736210    4470 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736213    4470 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736215    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736217    4470 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736220    4470 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736222    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736225    4470 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:59:07.760829 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736228    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736232    4470 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736237    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736240    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736243    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736247    4470 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736249    4470 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736258    4470 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736261    4470 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736264    4470 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736268    4470 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736272    4470 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736275    4470 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736278    4470 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736280    4470 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736283    4470 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736285    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736288    4470 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736291    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:59:07.761597 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736294    4470 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736297    4470 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736300    4470 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736302    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736305    4470 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736308    4470 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736310    4470 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736313    4470 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736315    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736317    4470 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736320    4470 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736322    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736325    4470 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736327    4470 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736330    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736332    4470 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736335    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736337    4470 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736340    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736342    4470 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:59:07.762431 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736345    4470 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:59:07.762162 ip-10-0-129-24 systemd[1]: Started Kubernetes Kubelet.
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736348    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736350    4470 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736353    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736355    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736358    4470 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736360    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736362    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736365    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736367    4470 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736370    4470 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736372    4470 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736375    4470 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736377    4470 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736380    4470 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736382    4470 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736384    4470 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736387    4470 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736389    4470 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736392    4470 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:59:07.763357 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736394    4470 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736396    4470 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736399    4470 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736401    4470 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736404    4470 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736407    4470 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736409    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736412    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736414    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736417    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736419    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736422    4470 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736424    4470 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736427    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.736451    4470 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736484    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 06:59:07.764003 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736488    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736491    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736493    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736496    4470 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736498    4470 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736501    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736504    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736506    4470 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736509    4470 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736512    4470 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736514    4470 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736517    4470 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736519    4470 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736521    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736524    4470 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736526    4470 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736529    4470 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736531    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736533    4470 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736536    4470 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 06:59:07.764547 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736538    4470 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736541    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736543    4470 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736545    4470 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736548    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736550    4470 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736553    4470 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736555    4470 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736558    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736560    4470 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736563    4470 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736565    4470 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736568    4470 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736570    4470 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736573    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736575    4470 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736578    4470 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736580    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736582    4470 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 06:59:07.765090 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736588    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736592    4470 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736596    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736599    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736601    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736604    4470 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736607    4470 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736609    4470 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736612    4470 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736614    4470 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736618    4470 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736622    4470 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736625    4470 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736628    4470 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736631    4470 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736634    4470 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736636    4470 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736639    4470 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736642    4470 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 06:59:07.765615 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736644    4470 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736647    4470 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736649    4470 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736651    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736654    4470 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736657    4470 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736660    4470 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736662    4470 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736665    4470 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736667    4470 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736670    4470 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736672    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736675    4470 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736678    4470 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736681    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736683    4470 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736685    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736688    4470 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736690    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736693    4470 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 06:59:07.766212 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736695    4470 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736698    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736700    4470 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736702    4470 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736705    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736707    4470 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736710    4470 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736715    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736718    4470 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736721    4470 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736723    4470 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736726    4470 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736728    4470 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736731    4470 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736733    4470 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736735    4470 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736738    4470 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736740    4470 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736743    4470 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736746    4470 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 06:59:07.766786 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736748    4470 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736750    4470 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736753    4470 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736755    4470 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736758    4470 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736763    4470 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:07.736765    4470 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.736814    4470 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.739008    4470 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.739055    4470 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.739252    4470 server.go:1014] "Starting client certificate rotation"
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.739381    4470 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.739424    4470 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 06:53:27 +0000 UTC" deadline="2028-06-16 22:53:43.441522462 +0000 UTC"
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.739450    4470 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14199h54m35.702073801s"
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.739712    4470 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.740755    4470 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 06:59:07.767358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.742239    4470 log.go:25] "Validated CRI v1 runtime API"
Nov 03 06:59:07.767942 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.743751    4470 log.go:25] "Validated CRI v1 image API"
Nov 03 06:59:07.767942 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.744465    4470 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 06:59:07.767942 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.745910    4470 fs.go:135] Filesystem UUIDs: map[1de08faf-a56b-48b8-9e52-49d5773e638d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9fbc50de-2a18-4d61-bf4c-9a46a3eae5b1:/dev/nvme0n1p4]
Nov 03 06:59:07.768036 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.745929    4470 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14ea2886b367f03e10c8a977483adc77764c9d22c5908dcfd1b920fe13019cfd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14ea2886b367f03e10c8a977483adc77764c9d22c5908dcfd1b920fe13019cfd/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dd239fb9366ccf33b06de4b3aba8fd62060181960794ae44e434b68d3e23d6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dd239fb9366ccf33b06de4b3aba8fd62060181960794ae44e434b68d3e23d6e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2826d0dc2418bdd741086e4fd1a1c5a293213e4647d24d5000c47799a1098e54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2826d0dc2418bdd741086e4fd1a1c5a293213e4647d24d5000c47799a1098e54/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c147a484bb1e1327c07a9e5d323904fbcc70a215e17ba690b46198eaf8855b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c147a484bb1e1327c07a9e5d323904fbcc70a215e17ba690b46198eaf8855b0/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b320f9d0777a26fea988bb6b747043a8bb056d60fb4a68eb9f07ccb50eee367/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b320f9d0777a26fea988bb6b747043a8bb056d60fb4a68eb9f07ccb50eee367/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/688b3cd0e94a268b40abd853ace4742279560e36d41986e06efb1656d7aed4eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/688b3cd0e94a268b40abd853ace4742279560e36d41986e06efb1656d7aed4eb/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71216bcbfdc6bb3860968ef41283611d25b43b64dfb52a25eaf5e258e251bfcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71216bcbfdc6bb3860968ef41283611d25b43b64dfb52a25eaf5e258e251bfcf/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8aedb969ece40c2805d597a3c2bb1334bb716722787f1e34469c68ea283b7704/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8aedb969ece40c2805d597a3c2bb1334bb716722787f1e34469c68ea283b7704/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a85e6df9a6ae53c3e7a1ad0586ef4fec6423d066c1fafba595a7514728697b17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a85e6df9a6ae53c3e7a1ad0586ef4fec6423d066c1fafba595a7514728697b17/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/add3c286ab668399a7118f1f8d30168cffcd5d439f8abba785e4a016a4b0c4d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/add3c286ab668399a7118f1f8d30168cffcd5d439f8abba785e4a016a4b0c4d4/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4aa3efb81e74ecbdd6da5dc69c924b7233bd5c6c72c913692f3f040eed58b17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4aa3efb81e74ecbdd6da5dc69c924b7233bd5c6c72c913692f3f040eed58b17/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8a36f1128dcc6662802931f6d8de793885e2bb329f1b262eb6e81bb450f85f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8a36f1128dcc6662802931f6d8de793885e2bb329f1b262eb6e81bb450f85f2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c62863da801620bda30b9d21be344d93340900bb64fbc9bdaa8db16212f82e31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c62863da801620bda30b9d21be344d93340900bb64fbc9bdaa8db16212f82e31/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9703783abab36c8ee8998cab1e352c8eea7c4b8c85bc262ab1ade211017da9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9703783abab36c8ee8998cab1e352c8eea7c4b8c85bc262ab1ade211017da9a/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0260ec75b016e0862207d2a0c8467c296d6aab04a6ca3bf6560b514f4214a47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0260ec75b016e0862207d2a0c8467c296d6aab04a6ca3bf6560b514f4214a47/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/017d594a-16ee-456f-8b9f-2fecc89acb9b/volumes/kubernetes.io~projected/kube-api-access-9zgtp:{mountpoint:/var/lib/kubelet/pods/017d594a-16ee-456f-8b9f-2fecc89acb9b/volumes/kubernetes.io~projected/kube-api-access-9zgtp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01ab7be9-13bf-4798-bcb0-44ca9923ce0c/volumes/kubernetes.io~projected/kube-api-access-n7bgl:{mountpoint:/var/lib/kubelet/pods/01ab7be9-13bf-4798-bcb0-44ca9923ce0c/volumes/kubernetes.io~projected/kube-api-access-n7bgl major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/021ef7d9-672a-4953-bb37-3ca4c271c7ca/volumes/kubernetes.io~projected/kube-api-access-scr95:{mountpoint:/var/lib/kubelet/pods/021ef7d9-672a-4953-bb37-3ca4c271c7ca/volumes/kubernetes.io~projected/kube-api-access-scr95 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03645044-2dfb-4a33-8a26-c3c5684cbe09/volumes/kubernetes.io~projected/kube-api-access-bfk5v:{mountpoint:/var/lib/kubelet/pods/03645044-2dfb-4a33-8a26-c3c5684cbe09/volumes/kubernetes.io~projected/kube-api-access-bfk5v major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03645044-2dfb-4a33-8a26-c3c5684cbe09/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/03645044-2dfb-4a33-8a26-c3c5684cbe09/volumes/kubernetes.io~secret/signing-key major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cee886c-6e9d-431b-9cda-0f8cd618cf2b/volumes/kubernetes.io~projected/kube-api-access-v754j:{mountpoint:/var/lib/kubelet/pods/2cee886c-6e9d-431b-9cda-0f8cd618cf2b/volumes/kubernetes.io~projected/kube-api-access-v754j major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cee886c-6e9d-431b-9cda-0f8cd618cf2b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2cee886c-6e9d-431b-9cda-0f8cd618cf2b/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c56f558-2833-421b-a2a4-fd023736ea7c/volumes/kubernetes.io~projected/kube-api-access-8b57d:{mountpoint:/var/lib/kubelet/pods/3c56f558-2833-421b-a2a4-fd023736ea7c/volumes/kubernetes.io~projected/kube-api-access-8b57d major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cea0af3-b692-43ed-8c86-df442fd5378c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5cea0af3-b692-43ed-8c86-df442fd5378c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cea0af3-b692-43ed-8c86-df442fd5378c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5cea0af3-b692-43ed-8c86-df442fd5378c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cea0af3-b692-43ed-8c86-df442fd5378c/volumes/kubernetes.io~projected/kube-api-access-bx8vm:{mountpoint:/var/lib/kubelet/pods/5cea0af3-b692-43ed-8c86-df442fd5378c/volumes/kubernetes.io~projected/kube-api-access-bx8vm major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bcf499f-e3d7-4146-89f8-74e2ae1685e3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7bcf499f-e3d7-4146-89f8-74e2ae1685e3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bcf499f-e3d7-4146-89f8-74e2ae1685e3/volumes/kubernetes.io~projected/kube-api-access-jn9q6:{mountpoint:/var/lib/kubelet/pods/7bcf499f-e3d7-4146-89f8-74e2ae1685e3/volumes/kubernetes.io~projected/kube-api-access-jn9q6 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bcf499f-e3d7-4146-89f8-74e2ae1685e3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7bcf499f-e3d7-4146-89f8-74e2ae1685e3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d04cba6-64ad-4a4e-8045-3c91d3237a14/volumes/kubernetes.io~projected/kube-api-access-r9lfh:{mountpoint:/var/lib/kubelet/pods/7d04cba6-64ad-4a4e-8045-3c91d3237a14/volumes/kubernetes.io~projected/kube-api-access-r9lfh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92fece83-25d4-4543-8489-67583d71ceff/volumes/kubernetes.io~projected/kube-api-access-2b46q:{mountpoint:/var/lib/kubelet/pods/92fece83-25d4-4543-8489-67583d71ceff/volumes/kubernetes.io~projected/kube-api-access-2b46q major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c46b893-7db9-4a89-a701-12fa631394fa/volumes/kubernetes.io~projected/kube-api-access-6kslf:{mountpoint:/var/lib/kubelet/pods/9c46b893-7db9-4a89-a701-12fa631394fa/volumes/kubernetes.io~projected/kube-api-access-6kslf major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5f4ea98-f511-4031-8b8b-7d057e1433cd/volumes/kubernetes.io~projected/kube-api-access-jrkvw:{mountpoint:/var/lib/kubelet/pods/a5f4ea98-f511-4031-8b8b-7d057e1433cd/volumes/kubernetes.io~projected/kube-api-access-jrkvw major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab43a143-464a-4155-98e7-3f7c480aeca6/volumes/kubernetes.io~projected/kube-api-access-b44wp:{mountpoint:/var/lib/kubelet/pods/ab43a143-464a-4155-98e7-3f7c480aeca6/volumes/kubernetes.io~projected/kube-api-access-b44wp major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af5953b4-e51d-433b-9860-6b077061f77c/volumes/kubernetes.io~projected/kube-api-access-2d5mj:{mountpoint:/var/lib/kubelet/pods/af5953b4-e51d-433b-9860-6b077061f77c/volumes/kubernetes.io~projected/kube-api-access-2d5mj major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3/volumes/kubernetes.io~projected/kube-api-access-bljw5:{mountpoint:/var/lib/kubelet/pods/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3/volumes/kubernetes.io~projected/kube-api-access-bljw5 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5aebe2f-412e-4d2a-b644-7fd13bb62e07/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b5aebe2f-412e-4d2a-b644-7fd13bb62e07/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf458915-e617-486b-8b9f-9019d09ca398/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/cf458915-e617-486b-8b9f-9019d09ca398/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:218 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/be84041ceba56c90b83db58b16d2c80e998d5b04d4c13e073948ecbb94104fea/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/87a9ef8f5725b6ce32c2bda615ba3dc0d080279c4faafe64d1d9c3d2b79c5a20/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a9c4199e5d20ee7430b6663af337b5dc17e597ad6974d1d858ee68237cf19259/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/ebbcf73dfc904c2d5885d7159c796c0b2753ac017d3b728e5109305671d3517b/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b81ecd05a1af9413fae08eb9a053b7a70015aa527dc9a22bc14ee5184e926046/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/daa58bb52e5cc60f464a5b7431613b2058f5abbe96088c09f91c3d2229f4acf5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/774dc56064ae5ec61bf4648cfec36dcd20aa94bed609c51c68e73b061a0f74b9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d26a73c56fd0ebd729939d7e71f16a2421f6edaf247133d4e24b05f65d862192/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/8486bb8868a41c63cd2d1e303fd72c620b78e6cadf75afbf9eebfcb73c8bc182/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/219f17aee339831808b828ba420d69194981d27b6609a9d61744eafc37f934ab/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/d83faf3c04aaaa47182cfafa3bd15ee8547406e37171e677a7613904724c9dc9/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/5bc44b19830169f990602673cf19c707ec58a1ad2356528cb039c5aa426f342b/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/96f920819aa32e5dccb8a909e6f626066b1bf3fd8d080eec210baa3adf392497/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/31147e7e8f9836197a3dc180fe3fe9e27b08684d3d31f64fc24a18c1df15fcbe/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6b8d207e9a2e020f5b260389c4ea19d3410b779158a24d8159427711aa821f0e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/59bd807b3642b1a235aaa328ec9c89247a63c432be5c068cf07ef21c210ea227/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/24bde9aa10d89fb6a64c1023a70a480c15af273fef43d995c21e64b8e2182394/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/bdfd9b669b44a7f0048478d2d49baff6aaef2a56711418504efffaf899fee042/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f71595c9a3fec9890695d8d9f90085c30e3260b8a179ccf7773c13dfda93749e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e813c906baeac780562bff56990c54dda5ac19463e5a2b6b82929e2d9cc59c26/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b98756c394b6af0827c663870fe4b7fee6121a5c46e59fa64fdec469f405b3c4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/f335c0b055d73dcecbbc7c21db8fc2f59121cbc35446cf953192f3101b3df2f7/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/046ae228aad5e3e0db4dbc3b88c027ae4feb2bbcba7e80811322962347245f09/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/7a9bc559b154f1443285a96414b36441f89ac0947c3b3ac9e074abbcd07b2774/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/2ca459fe3ab07aec352fd2cb60b42aa017f65febe4d785352b9a435b7f8edbc3/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/cf9ab4e68660cf2caf7b2068c796cd531a3bfd056165adf1696a9545161eaf41/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/c1a3808e664a186ae5650ce7d9221a8cc2ed262befba4ffba003e62b32eb8f14/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/be29f978af5c7d7a7d5e7ba2a201598bf36a0604ece0ff95289036d5de56e79f/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/04028e19e1af9cbe6fd1f87ddb2d7c6ec64b3a6af266c1a000c46854eed080e0/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/cf6e8fe9d641563eb111283956430662f7d54c19893cfc1f2fc8e926d95a3d5f/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/0e9a6307721638cd180181c66c007c80abd205efd9fa4becf385f6113d292795/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/ed7f08c4d31e8e7dd75ff684f63cf6ed09274754ca2686ef4cea2b363a206ded/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8f207c7e26dc90083645fed80fe337b235cdd53f1dfb780e66b159d61bd6ae84/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/3ded9374bd3028a9eaee8ed011161d0bf4860f33bf2a2ce6ef673d9de1a6ce02/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/b586ae57ce0cf564f0c78caa0bca85434e9390fe5712fef454d93e5e570773d6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/354f7b4ecea3c82d364821797962320cd907500e02195dcf67fb6fdcffe9c4d5/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/89834e9074c60383823c2de38fe49bb38fcf2a840618543ebcc9831b93d081ff/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/1c44e6668d53735ce1aa8c9f7b4bdf9f8259811fba52e6a59f06f27588c29741/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/66dc9c00526a8a293172004e770a2d89dde2fba4ebd68f0dea8c243cd4ccc3ab/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/0535496fa72aa4dad3d74b58d175b7bf7afa47f34ba0739019825ad8b9dffbfe/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/666348986a3d2e8875e3574e9d38375630ab2158235c9ee3fd857c52f13dc6ff/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.753032    4470 manager.go:217] Machine: {Timestamp:2026-11-03 06:59:07.752055746 +0000 UTC m=+0.084424730 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097092 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d4197316f2349a84ccf421a7de303 SystemUUID:ec2d4197-316f-2349-a84c-cf421a7de303 BootID:230379fa-dd6d-41c6-bbd5-6f2683fc7338 Filesystems:[{Device:/var/lib/kubelet/pods/5cea0af3-b692-43ed-8c86-df442fd5378c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03645044-2dfb-4a33-8a26-c3c5684cbe09/volumes/kubernetes.io~projected/kube-api-access-bfk5v DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7bcf499f-e3d7-4146-89f8-74e2ae1685e3/volumes/kubernetes.io~projected/kube-api-access-jn9q6 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2826d0dc2418bdd741086e4fd1a1c5a293213e4647d24d5000c47799a1098e54/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2cee886c-6e9d-431b-9cda-0f8cd618cf2b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a85e6df9a6ae53c3e7a1ad0586ef4fec6423d066c1fafba595a7514728697b17/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5aebe2f-412e-4d2a-b644-7fd13bb62e07/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab43a143-464a-4155-98e7-3f7c480aeca6/volumes/kubernetes.io~projected/kube-api-access-b44wp DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5cea0af3-b692-43ed-8c86-df442fd5378c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92fece83-25d4-4543-8489-67583d71ceff/volumes/kubernetes.io~projected/kube-api-access-2b46q DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2cee886c-6e9d-431b-9cda-0f8cd618cf2b/volumes/kubernetes.io~projected/kube-api-access-v754j DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/688b3cd0e94a268b40abd853ace4742279560e36d41986e06efb1656d7aed4eb/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cf458915-e617-486b-8b9f-9019d09ca398/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3/volumes/kubernetes.io~projected/kube-api-access-bljw5 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8aedb969ece40c2805d597a3c2bb1334bb716722787f1e34469c68ea283b7704/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4aa3efb81e74ecbdd6da5dc69c924b7233bd5c6c72c913692f3f040eed58b17/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af5953b4-e51d-433b-9860-6b077061f77c/volumes/kubernetes.io~projected/kube-api-access-2d5mj DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0260ec75b016e0862207d2a0c8467c296d6aab04a6ca3bf6560b514f4214a47/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9c46b893-7db9-4a89-a701-12fa631394fa/volumes/kubernetes.io~projected/kube-api-access-6kslf DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14ea2886b367f03e10c8a977483adc77764c9d22c5908dcfd1b920fe13019cfd/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03645044-2dfb-4a33-8a26-c3c5684cbe09/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9703783abab36c8ee8998cab1e352c8eea7c4b8c85bc262ab1ade211017da9a/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/5cea0af3-b692-43ed-8c86-df442fd5378c/volumes/kubernetes.io~projected/kube-api-access-bx8vm DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/add3c286ab668399a7118f1f8d30168cffcd5d439f8abba785e4a016a4b0c4d4/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b320f9d0777a26fea988bb6b747043a8bb056d60fb4a68eb9f07ccb50eee367/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7bcf499f-e3d7-4146-89f8-74e2ae1685e3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c62863da801620bda30b9d21be344d93340900bb64fbc9bdaa8db16212f82e31/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5f4ea98-f511-4031-8b8b-7d057e1433cd/volumes/kubernetes.io~projected/kube-api-access-jrkvw DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c147a484bb1e1327c07a9e5d323904fbcc70a215e17ba690b46198eaf8855b0/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d04cba6-64ad-4a4e-8045-3c91d3237a14/volumes/kubernetes.io~projected/kube-api-access-r9lfh DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c56f558-2833-421b-a2a4-fd023736ea7c/volumes/kubernetes.io~projected/kube-api-access-8b57d DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71216bcbfdc6bb3860968ef41283611d25b43b64dfb52a25eaf5e258e251bfcf/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8a36f1128dcc6662802931f6d8de793885e2bb329f1b262eb6e81bb450f85f2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dd239fb9366ccf33b06de4b3aba8fd62060181960794ae44e434b68d3e23d6e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/017d594a-16ee-456f-8b9f-2fecc89acb9b/volumes/kubernetes.io~projected/kube-api-access-9zgtp DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/021ef7d9-672a-4953-bb37-3ca4c271c7ca/volumes/kubernetes.io~projected/kube-api-access-scr95 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7bcf499f-e3d7-4146-89f8-74e2ae1685e3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01ab7be9-13bf-4798-bcb0-44ca9923ce0c/volumes/kubernetes.io~projected/kube-api-access-n7bgl DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3c147a484bb1e13 MacAddress:82:59:48:c9:58:d8 Speed:10000 Mtu:8901} {Name:688b3cd0e94a268 MacAddress:aa:ce:8a:36:14:02 Speed:10000 Mtu:8901} {Name:add3c286ab66839 MacAddress:8e:e1:61:23:b4:0f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:01:98:a4:a1:71 Speed:0 Mtu:9001} {Name:br-int MacAddress:b2:dd:2d:43:7c:be Speed:0 Mtu:8901} {Name:c9703783abab36c MacAddress:9e:83:27:d3:26:3e Speed:10000 Mtu:8901} {Name:d0260ec75b016e0 MacAddress:aa:a1:4f:44:2f:22 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:01:98:a4:a1:71 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c2:7e:c2:ce:62:14 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ee:ab:dc:ab:3a:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.753759    4470 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.753821    4470 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754101    4470 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754152    4470 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-24.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754279    4470 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754287    4470 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754300    4470 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754312    4470 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754398    4470 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754553    4470 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754668    4470 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754706    4470 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754722    4470 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754737    4470 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754746    4470 kubelet.go:405] "Adding apiserver pod source"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.754754    4470 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.755490    4470 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.755502    4470 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.755989    4470 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.757768    4470 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.757776    4470 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.757868    4470 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758047    4470 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758074    4470 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758176    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758187    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758194    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758199    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758205    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758211    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758216    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758221    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758229    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758235    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758249    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758258    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758281    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.758287    4470 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.761554    4470 server.go:1263] "Started kubelet"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.761632    4470 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.761710    4470 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.761764    4470 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.762319    4470 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.763522    4470 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 06:59:07.768240 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.765303    4470 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 06:59:07.771489 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.771467    4470 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 06:59:07.771715 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.771671    4470 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 06:53:29 +0000 UTC" deadline="2028-06-08 19:15:53.777407384 +0000 UTC"
Nov 03 06:59:07.771783 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.771715    4470 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14004h16m46.005695633s"
Nov 03 06:59:07.771783 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:07.771758    4470 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found"
Nov 03 06:59:07.771903 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.771799    4470 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 06:59:07.771903 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.771809    4470 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 06:59:07.771991 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.771923    4470 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775021    4470 factory.go:55] Registering systemd factory
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775043    4470 factory.go:223] Registration of the systemd container factory successfully
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775297    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bcf499f-e3d7-4146-89f8-74e2ae1685e3" volumeName="kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775330    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5f4ea98-f511-4031-8b8b-7d057e1433cd" volumeName="kubernetes.io/projected/a5f4ea98-f511-4031-8b8b-7d057e1433cd-kube-api-access-jrkvw" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775343    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5aebe2f-412e-4d2a-b644-7fd13bb62e07" volumeName="kubernetes.io/configmap/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-konnectivity-ca" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775354    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92fece83-25d4-4543-8489-67583d71ceff" volumeName="kubernetes.io/configmap/92fece83-25d4-4543-8489-67583d71ceff-iptables-alerter-script" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775367    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" volumeName="kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-binary-copy" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775378    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="021ef7d9-672a-4953-bb37-3ca4c271c7ca" volumeName="kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cni-binary-copy" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775389    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03645044-2dfb-4a33-8a26-c3c5684cbe09" volumeName="kubernetes.io/configmap/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-cabundle" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775399    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cee886c-6e9d-431b-9cda-0f8cd618cf2b" volumeName="kubernetes.io/projected/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-kube-api-access-v754j" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775411    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cea0af3-b692-43ed-8c86-df442fd5378c" volumeName="kubernetes.io/projected/5cea0af3-b692-43ed-8c86-df442fd5378c-kube-api-access-bx8vm" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775422    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5f4ea98-f511-4031-8b8b-7d057e1433cd" volumeName="kubernetes.io/configmap/a5f4ea98-f511-4031-8b8b-7d057e1433cd-config-volume" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775459    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" volumeName="kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775471    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="021ef7d9-672a-4953-bb37-3ca4c271c7ca" volumeName="kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-daemon-config" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775482    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03645044-2dfb-4a33-8a26-c3c5684cbe09" volumeName="kubernetes.io/secret/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-key" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775492    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cee886c-6e9d-431b-9cda-0f8cd618cf2b" volumeName="kubernetes.io/secret/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-cert" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775505    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bcf499f-e3d7-4146-89f8-74e2ae1685e3" volumeName="kubernetes.io/projected/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-kube-api-access-jn9q6" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775521    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92fece83-25d4-4543-8489-67583d71ceff" volumeName="kubernetes.io/projected/92fece83-25d4-4543-8489-67583d71ceff-kube-api-access-2b46q" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775537    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="017d594a-16ee-456f-8b9f-2fecc89acb9b" volumeName="kubernetes.io/configmap/017d594a-16ee-456f-8b9f-2fecc89acb9b-serviceca" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775554    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c56f558-2833-421b-a2a4-fd023736ea7c" volumeName="kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775565    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c56f558-2833-421b-a2a4-fd023736ea7c" volumeName="kubernetes.io/empty-dir/3c56f558-2833-421b-a2a4-fd023736ea7c-archives-path" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775576    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5f4ea98-f511-4031-8b8b-7d057e1433cd" volumeName="kubernetes.io/empty-dir/a5f4ea98-f511-4031-8b8b-7d057e1433cd-tmp-dir" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775587    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5aebe2f-412e-4d2a-b644-7fd13bb62e07" volumeName="kubernetes.io/secret/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-agent-certs" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775597    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf458915-e617-486b-8b9f-9019d09ca398" volumeName="kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775608    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="017d594a-16ee-456f-8b9f-2fecc89acb9b" volumeName="kubernetes.io/projected/017d594a-16ee-456f-8b9f-2fecc89acb9b-kube-api-access-9zgtp" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775620    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" volumeName="kubernetes.io/projected/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-kube-api-access-n7bgl" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775630    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03645044-2dfb-4a33-8a26-c3c5684cbe09" volumeName="kubernetes.io/projected/03645044-2dfb-4a33-8a26-c3c5684cbe09-kube-api-access-bfk5v" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775642    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c56f558-2833-421b-a2a4-fd023736ea7c" volumeName="kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-service-ca-bundle" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775653    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bcf499f-e3d7-4146-89f8-74e2ae1685e3" volumeName="kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-config" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775662    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c46b893-7db9-4a89-a701-12fa631394fa" volumeName="kubernetes.io/empty-dir/9c46b893-7db9-4a89-a701-12fa631394fa-tmp-dir" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775678    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" volumeName="kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775689    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c56f558-2833-421b-a2a4-fd023736ea7c" volumeName="kubernetes.io/projected/3c56f558-2833-421b-a2a4-fd023736ea7c-kube-api-access-8b57d" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775700    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d04cba6-64ad-4a4e-8045-3c91d3237a14" volumeName="kubernetes.io/projected/7d04cba6-64ad-4a4e-8045-3c91d3237a14-kube-api-access-r9lfh" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775717    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab43a143-464a-4155-98e7-3f7c480aeca6" volumeName="kubernetes.io/projected/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-api-access-b44wp" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775728    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="021ef7d9-672a-4953-bb37-3ca4c271c7ca" volumeName="kubernetes.io/projected/021ef7d9-672a-4953-bb37-3ca4c271c7ca-kube-api-access-scr95" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775740    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cea0af3-b692-43ed-8c86-df442fd5378c" volumeName="kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-tuned" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775751    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cea0af3-b692-43ed-8c86-df442fd5378c" volumeName="kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-tmp" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775762    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bcf499f-e3d7-4146-89f8-74e2ae1685e3" volumeName="kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-env-overrides" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775773    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bcf499f-e3d7-4146-89f8-74e2ae1685e3" volumeName="kubernetes.io/secret/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775783    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c46b893-7db9-4a89-a701-12fa631394fa" volumeName="kubernetes.io/projected/9c46b893-7db9-4a89-a701-12fa631394fa-kube-api-access-6kslf" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775794    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab43a143-464a-4155-98e7-3f7c480aeca6" volumeName="kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775804    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab43a143-464a-4155-98e7-3f7c480aeca6" volumeName="kubernetes.io/empty-dir/ab43a143-464a-4155-98e7-3f7c480aeca6-data-volume" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775815    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af5953b4-e51d-433b-9860-6b077061f77c" volumeName="kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775826    4470 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b31cdbad-0b70-43c3-89c8-3fce65ba4ec3" volumeName="kubernetes.io/projected/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kube-api-access-bljw5" seLinuxMountContext=""
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775837    4470 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 06:59:07.776165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.775844    4470 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 06:59:07.779586 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.777960    4470 factory.go:153] Registering CRI-O factory
Nov 03 06:59:07.779586 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.777972    4470 factory.go:223] Registration of the crio container factory successfully
Nov 03 06:59:07.779586 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.778050    4470 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 06:59:07.779586 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.778071    4470 factory.go:103] Registering Raw factory
Nov 03 06:59:07.779586 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.778084    4470 manager.go:1199] Started watching for new ooms in manager
Nov 03 06:59:07.779586 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.778411    4470 manager.go:319] Starting recovery of all containers
Nov 03 06:59:07.796861 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.796818    4470 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 06:59:07.798461 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.798417    4470 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 06:59:07.798461 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.798438    4470 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 06:59:07.798629 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.798469    4470 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 06:59:07.798629 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:07.798514    4470 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 06:59:07.807210 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.807183    4470 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="daba2f72354958cd63dfcb2da3f2a0e8e3a4b88fbd5def0ee63f81b98480cc43" exitCode=0
Nov 03 06:59:07.807210 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.807208    4470 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="27532d715251c5d1588e34c3fb14bb004975a99d4e2833a1cef166a88db2412f" exitCode=0
Nov 03 06:59:07.807334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.807218    4470 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="a15eb34ea67d369ceb609d372f03ab59e73ec2d37961eca1e677df59a04d1232" exitCode=0
Nov 03 06:59:07.807334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.807227    4470 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="9e8eb59023191f53ae9a10cab458908ec93a9f411bddc8919df4ef06cf86e0d6" exitCode=0
Nov 03 06:59:07.807334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.807235    4470 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="f86a46f6ee2a7f9ee17d8dda4ab7111c497e3fba4259b23f0f886c104d12b1d9" exitCode=0
Nov 03 06:59:07.807334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.807246    4470 generic.go:356] "Generic (PLEG): container finished" podID="01ab7be9-13bf-4798-bcb0-44ca9923ce0c" containerID="4ab8753f067605fe0e64790646f78bd3ef9791974330de4fd81e64fd0acac7a8" exitCode=0
Nov 03 06:59:07.840104 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.840073    4470 generic.go:356] "Generic (PLEG): container finished" podID="529279bc3624b3843e9992a4e421e883" containerID="625a8dcd2ee4a3a0750f65eab1b256794dac1d86a8dc048d2726c0dc3ee050e9" exitCode=0
Nov 03 06:59:07.848063 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.847997    4470 manager.go:324] Recovery completed
Nov 03 06:59:07.861203 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.861184    4470 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:59:07.862218 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.862201    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:59:07.862293 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.862230    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:59:07.862293 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.862241    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:59:07.863550 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863526    4470 cpu_manager.go:226] "Starting" policy="none"
Nov 03 06:59:07.863550 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863547    4470 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 06:59:07.863658 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863553    4470 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:59:07.863658 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863565    4470 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 06:59:07.863719 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863673    4470 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 06:59:07.863719 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863682    4470 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 06:59:07.863719 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863708    4470 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 06:59:07.863719 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863713    4470 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 06:59:07.863719 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863718    4470 policy_none.go:50] "Start"
Nov 03 06:59:07.863871 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863726    4470 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 06:59:07.863871 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863735    4470 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 06:59:07.863871 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863853    4470 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 06:59:07.863871 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863859    4470 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 06:59:07.863871 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.863864    4470 policy_none.go:44] "Start"
Nov 03 06:59:07.864326 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.864304    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:59:07.864409 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.864335    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:59:07.864409 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.864350    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:59:07.867928 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.867903    4470 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 06:59:07.871228 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.871214    4470 manager.go:341] "Starting Device Plugin manager"
Nov 03 06:59:07.871295 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:07.871258    4470 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 06:59:07.871295 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.871271    4470 server.go:87] "Starting device plugin registration server"
Nov 03 06:59:07.871582 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.871571    4470 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 06:59:07.871627 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.871585    4470 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 06:59:07.871692 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.871667    4470 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 06:59:07.871837 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.871802    4470 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 06:59:07.871837 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.871846    4470 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 06:59:07.871939 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.871853    4470 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 06:59:07.871988 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.871912    4470 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T06:59:07.871836371Z","UUID":"10e5b1d9-5f79-488d-88a9-6a94b70323d3","Handler":null,"Name":"","Endpoint":""}
Nov 03 06:59:07.873224 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:07.873185    4470 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 06:59:07.873325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.873233    4470 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 06:59:07.873325 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:07.873246    4470 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-24.us-west-1.compute.internal\" not found"
Nov 03 06:59:07.873325 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.873251    4470 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 06:59:07.899624 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.899128    4470 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"]
Nov 03 06:59:07.899997 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.899974    4470 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:59:07.901193 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.901153    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:59:07.901299 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.901197    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:59:07.901299 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.901217    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:59:07.901618 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.901603    4470 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:59:07.901883 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.901870    4470 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:59:07.902846 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.902827    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:59:07.902907 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.902859    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:59:07.902907 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.902870    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:59:07.902907 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.902887    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:59:07.902907 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.902906    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:59:07.903028 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.902916    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:59:07.903060 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:07.903047    4470 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:07.903126 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.903098    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal" event={"ID":"2e6f5344ea2747ae378e95b2b7a4fbd6","Type":"ContainerStarted","Data":"d35b0d980303afb0a6fd3a68f6b854ddd056b00f4831f7aac4a51980b9f2c97b"}
Nov 03 06:59:07.903182 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.903159    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal" event={"ID":"2e6f5344ea2747ae378e95b2b7a4fbd6","Type":"ContainerStarted","Data":"a85e6df9a6ae53c3e7a1ad0586ef4fec6423d066c1fafba595a7514728697b17"}
Nov 03 06:59:07.903214 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.903179    4470 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:59:07.903258 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.903180    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal" event={"ID":"529279bc3624b3843e9992a4e421e883","Type":"ContainerStarted","Data":"230a7af561897fa89adfdaf79248eb030d57d740f38bd90efd930119f32fe5b7"}
Nov 03 06:59:07.903289 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.903276    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal" event={"ID":"529279bc3624b3843e9992a4e421e883","Type":"ContainerDied","Data":"625a8dcd2ee4a3a0750f65eab1b256794dac1d86a8dc048d2726c0dc3ee050e9"}
Nov 03 06:59:07.903316 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.903308    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal" event={"ID":"529279bc3624b3843e9992a4e421e883","Type":"ContainerStarted","Data":"1dd239fb9366ccf33b06de4b3aba8fd62060181960794ae44e434b68d3e23d6e"}
Nov 03 06:59:07.903854 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.903832    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:59:07.903893 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.903862    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:59:07.903893 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.903877    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:59:07.903998 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:07.903987    4470 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-24.us-west-1.compute.internal\" not found" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:07.969196 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.969170    4470 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 06:59:07.969987 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.969965    4470 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 06:59:07.970115 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.970017    4470 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 06:59:07.970115 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.970049    4470 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 06:59:07.972242 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.972222    4470 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 06:59:07.972302 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.972239    4470 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:07.973236 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.973217    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 06:59:07.973315 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.973245    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 06:59:07.973315 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.973255    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 06:59:07.973315 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:07.973289    4470 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.020971 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.020941    4470 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 06:59:08.061835 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.061804    4470 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 06:59:08.061952 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:08.061845    4470 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.061952 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.061863    4470 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.070021 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.069996    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.070125 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.070034    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.070125 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.070064    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e6f5344ea2747ae378e95b2b7a4fbd6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"2e6f5344ea2747ae378e95b2b7a4fbd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.118940 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.118876    4470 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.119039 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.118946    4470 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.119517 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.119501    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeNotReady"
Nov 03 06:59:08.119615 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.119529    4470 setters.go:546] "Node became not ready" node="ip-10-0-129-24.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T06:59:08Z","lastTransitionTime":"2026-11-03T06:59:08Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 03 06:59:08.121544 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.121530    4470 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 06:59:08.121593 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:08.121567    4470 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.137692 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.137668    4470 nodeinfomanager.go:471] existing CSINode "ip-10-0-129-24.us-west-1.compute.internal" is owned by different node (oldNodeID="cb2b699d-219b-44bd-96be-da6fd9629c5f", newNodeID=""), cleaning up...
Nov 03 06:59:08.170841 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.170817    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.170944 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.170844    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.170944 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.170861    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e6f5344ea2747ae378e95b2b7a4fbd6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"2e6f5344ea2747ae378e95b2b7a4fbd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.170944 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.170908    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e6f5344ea2747ae378e95b2b7a4fbd6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"2e6f5344ea2747ae378e95b2b7a4fbd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.170944 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.170931    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.170944 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.170936    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/529279bc3624b3843e9992a4e421e883-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" (UID: \"529279bc3624b3843e9992a4e421e883\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.754923 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.754895    4470 apiserver.go:52] "Watching apiserver"
Nov 03 06:59:08.842458 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.842430    4470 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.842588 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.842524    4470 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.861962 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.861942    4470 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 06:59:08.862454 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.862435    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-gqgl4","openshift-insights/periodic-gathering-l2dln-4d54l","kube-system/konnectivity-agent-9v5rw","kube-system/global-pull-secret-syncer-pw94d","openshift-multus/network-metrics-daemon-mq4wx","openshift-service-ca/service-ca-5fffc97bdf-6484x","openshift-image-registry/node-ca-pkxfm","openshift-ingress-canary/ingress-canary-6vxzv","openshift-network-operator/iptables-alerter-krd42","kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7","openshift-multus/multus-2dp6g","openshift-ovn-kubernetes/ovnkube-node-6w8lk","openshift-network-diagnostics/network-check-target-6bhzs","openshift-insights/insights-runtime-extractor-vprr7","openshift-dns/dns-default-pcxdc","openshift-dns/node-resolver-dxw74","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-xmqdp"]
Nov 03 06:59:08.866550 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.866521    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:59:08.870241 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.870220    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:08.870696 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.870678    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:08.873149 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.873118    4470 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 06:59:08.874426 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874411    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c56f558-2833-421b-a2a4-fd023736ea7c-archives-path\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:59:08.874546 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874434    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scr95\" (UniqueName: \"kubernetes.io/projected/021ef7d9-672a-4953-bb37-3ca4c271c7ca-kube-api-access-scr95\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.874546 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874452    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysconfig\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.874546 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874466    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-host\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.874546 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874484    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c46b893-7db9-4a89-a701-12fa631394fa-tmp-dir\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:59:08.874546 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874533    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kslf\" (UniqueName: \"kubernetes.io/projected/9c46b893-7db9-4a89-a701-12fa631394fa-kube-api-access-6kslf\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:59:08.874719 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874629    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c46b893-7db9-4a89-a701-12fa631394fa-tmp-dir\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:59:08.874719 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874645    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-etc-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.874719 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874671    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-kubelet-config\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:59:08.874719 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874707    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cni-binary-copy\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.874878 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874743    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-sys\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.874878 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874741    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c56f558-2833-421b-a2a4-fd023736ea7c-archives-path\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:59:08.874878 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874782    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-config\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.874878 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874837    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:08.874878 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874875    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f4ea98-f511-4031-8b8b-7d057e1433cd-tmp-dir\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:08.875079 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874904    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-var-lib-kubelet\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.875079 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874930    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/017d594a-16ee-456f-8b9f-2fecc89acb9b-serviceca\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:59:08.875079 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874953    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-kubelet\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.875079 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.874988    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-cert\") pod \"ingress-canary-6vxzv\" (UID: \"2cee886c-6e9d-431b-9cda-0f8cd618cf2b\") " pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:59:08.875079 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875041    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cnibin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.875079 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875043    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f4ea98-f511-4031-8b8b-7d057e1433cd-tmp-dir\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:08.875272 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875081    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-conf-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.875272 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875096    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-multus-certs\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.875272 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875127    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-conf\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.875272 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875177    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9q6\" (UniqueName: \"kubernetes.io/projected/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-kube-api-access-jn9q6\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.875272 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875213    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.875410 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875302    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-agent-certs\") pod \"konnectivity-agent-9v5rw\" (UID: \"b5aebe2f-412e-4d2a-b644-7fd13bb62e07\") " pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:59:08.875410 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875318    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-konnectivity-ca\") pod \"konnectivity-agent-9v5rw\" (UID: \"b5aebe2f-412e-4d2a-b644-7fd13bb62e07\") " pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:59:08.875410 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875344    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b46q\" (UniqueName: \"kubernetes.io/projected/92fece83-25d4-4543-8489-67583d71ceff-kube-api-access-2b46q\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:59:08.875410 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875388    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-netd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.875875 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875862    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-key\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:59:08.875910 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875895    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-k8s-cni-cncf-io\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.875939 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875911    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-hostroot\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.875967 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875953    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.875996 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.875975    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92fece83-25d4-4543-8489-67583d71ceff-iptables-alerter-script\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:59:08.876027 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876003    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-node-log\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.876071 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876057    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b57d\" (UniqueName: \"kubernetes.io/projected/3c56f558-2833-421b-a2a4-fd023736ea7c-kube-api-access-8b57d\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:59:08.876102 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876086    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-system-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.876147 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876114    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:59:08.876179 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876156    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-netns\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.876255 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876184    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7bgl\" (UniqueName: \"kubernetes.io/projected/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-kube-api-access-n7bgl\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.876255 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876212    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9lfh\" (UniqueName: \"kubernetes.io/projected/7d04cba6-64ad-4a4e-8045-3c91d3237a14-kube-api-access-r9lfh\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:59:08.876255 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876238    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgtp\" (UniqueName: \"kubernetes.io/projected/017d594a-16ee-456f-8b9f-2fecc89acb9b-kube-api-access-9zgtp\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:59:08.876358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876261    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.876358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876287    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-registration-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.876358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876309    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.876358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876332    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-socket-dir-parent\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.876358 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876354    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-lib-modules\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.876522 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876393    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:08.876522 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876429    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-sys-fs\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.876522 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876447    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-var-lib-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.876522 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876462    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.876522 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876483    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.876660 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876519    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f4ea98-f511-4031-8b8b-7d057e1433cd-config-volume\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:08.876660 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876549    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfk5v\" (UniqueName: \"kubernetes.io/projected/03645044-2dfb-4a33-8a26-c3c5684cbe09-kube-api-access-bfk5v\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:59:08.876660 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876596    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-kubelet\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.876660 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876633    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-etc-kubernetes\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.876785 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876680    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-systemd\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.876785 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876726    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab43a143-464a-4155-98e7-3f7c480aeca6-crio-socket\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:08.876785 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876768    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljw5\" (UniqueName: \"kubernetes.io/projected/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kube-api-access-bljw5\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.876865 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876793    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.876893 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876881    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-tuned\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.876924 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876896    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-tmp\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.876958 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876937    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-socket-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.876958 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876947    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-tmp\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.877041 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.876956    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-tuned\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.877041 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877000    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-slash\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.877149 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877041    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-systemd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.877149 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877072    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-ovn\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.877149 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877107    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-log-socket\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.877296 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877157    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-dbus\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:59:08.877296 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877183    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cnibin\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.877296 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877224    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.877296 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877253    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-run\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.877408 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877292    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:08.877408 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877338    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovn-node-metrics-cert\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.877408 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877364    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-service-ca-bundle\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:59:08.877408 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877402    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v754j\" (UniqueName: \"kubernetes.io/projected/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-kube-api-access-v754j\") pod \"ingress-canary-6vxzv\" (UID: \"2cee886c-6e9d-431b-9cda-0f8cd618cf2b\") " pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:59:08.877591 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877437    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-bin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.877591 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877453    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-kubernetes\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.877591 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877491    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-netns\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.877591 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877510    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-cabundle\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:59:08.877591 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877527    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8vm\" (UniqueName: \"kubernetes.io/projected/5cea0af3-b692-43ed-8c86-df442fd5378c-kube-api-access-bx8vm\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.877591 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877547    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5mj\" (UniqueName: \"kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj\") pod \"network-check-target-6bhzs\" (UID: \"af5953b4-e51d-433b-9860-6b077061f77c\") " pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:59:08.877591 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877570    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/017d594a-16ee-456f-8b9f-2fecc89acb9b-host\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:59:08.877815 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877615    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-bin\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.877815 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877687    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-script-lib\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.877815 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877741    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-multus\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.877815 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877766    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-daemon-config\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.877815 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877812    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:59:08.877949 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877837    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab43a143-464a-4155-98e7-3f7c480aeca6-data-volume\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:08.877949 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877877    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92fece83-25d4-4543-8489-67583d71ceff-host-slash\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:59:08.877949 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877908    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab43a143-464a-4155-98e7-3f7c480aeca6-data-volume\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:08.877949 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877918    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-trusted-ca-bundle\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:59:08.878055 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877955    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.878055 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.877988    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.878055 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.878010    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-modprobe-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.878055 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.878037    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b44wp\" (UniqueName: \"kubernetes.io/projected/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-api-access-b44wp\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:08.878202 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.878060    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-device-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.878202 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.878075    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c46b893-7db9-4a89-a701-12fa631394fa-hosts-file\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:59:08.878202 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.878093    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.878202 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.878123    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrkvw\" (UniqueName: \"kubernetes.io/projected/a5f4ea98-f511-4031-8b8b-7d057e1433cd-kube-api-access-jrkvw\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:08.878202 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.878168    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-system-cni-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.878202 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.878198    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-systemd-units\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.878363 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.878224    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-env-overrides\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.905032 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.905011    4470 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 06:59:08.905117 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:08.905057    4470 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.906663 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.906650    4470 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 06:59:08.906711 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:08.906687    4470 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-24.us-west-1.compute.internal"
Nov 03 06:59:08.917378 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.917351    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 06:59:08.917378 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.917352    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 06:59:08.917490 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.917352    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 06:59:08.917490 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.917410    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 06:59:08.917490 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.917436    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 06:59:08.917490 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.917437    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 06:59:08.918665 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.918651    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 06:59:08.918915 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.918900    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lr4sn\""
Nov 03 06:59:08.918951 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.918909    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tq2gc\""
Nov 03 06:59:08.918994 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.918982    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxfbc\""
Nov 03 06:59:08.921914 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.921898    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 06:59:08.921963 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.921911    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 06:59:08.921963 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.921903    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 06:59:08.921963 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.921947    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 06:59:08.922226 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.922211    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 06:59:08.922266 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.922224    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 06:59:08.922310 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.922281    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 06:59:08.922310 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.922281    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 06:59:08.922393 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.922316    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 06:59:08.922423 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.922393    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 06:59:08.922542 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.922528    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 06:59:08.922580 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.922539    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 06:59:08.922996 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.922982    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 06:59:08.923055 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.923001    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 06:59:08.923055 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.923012    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 06:59:08.925462 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.925438    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cni-binary-copy\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.925539 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.925500    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/017d594a-16ee-456f-8b9f-2fecc89acb9b-serviceca\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:59:08.925681 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.925663    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-konnectivity-ca\") pod \"konnectivity-agent-9v5rw\" (UID: \"b5aebe2f-412e-4d2a-b644-7fd13bb62e07\") " pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:59:08.925714 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.925701    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5aebe2f-412e-4d2a-b644-7fd13bb62e07-agent-certs\") pod \"konnectivity-agent-9v5rw\" (UID: \"b5aebe2f-412e-4d2a-b644-7fd13bb62e07\") " pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:59:08.925756 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.925706    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.926802 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.926784    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92fece83-25d4-4543-8489-67583d71ceff-iptables-alerter-script\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:59:08.927899 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.927883    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-script-lib\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.928935 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.928919    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-env-overrides\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.928987 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.928943    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.928987 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.928967    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.960816 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.960745    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8vm\" (UniqueName: \"kubernetes.io/projected/5cea0af3-b692-43ed-8c86-df442fd5378c-kube-api-access-bx8vm\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.961321 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.961295    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5mj\" (UniqueName: \"kubernetes.io/projected/af5953b4-e51d-433b-9860-6b077061f77c-kube-api-access-2d5mj\") pod \"network-check-target-6bhzs\" (UID: \"af5953b4-e51d-433b-9860-6b077061f77c\") " pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:59:08.962128 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.961469    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kslf\" (UniqueName: \"kubernetes.io/projected/9c46b893-7db9-4a89-a701-12fa631394fa-kube-api-access-6kslf\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:59:08.962128 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.961885    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrkvw\" (UniqueName: \"kubernetes.io/projected/a5f4ea98-f511-4031-8b8b-7d057e1433cd-kube-api-access-jrkvw\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:08.962128 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.961900    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scr95\" (UniqueName: \"kubernetes.io/projected/021ef7d9-672a-4953-bb37-3ca4c271c7ca-kube-api-access-scr95\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.962128 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.962099    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7bgl\" (UniqueName: \"kubernetes.io/projected/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-kube-api-access-n7bgl\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.962828 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.962805    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9lfh\" (UniqueName: \"kubernetes.io/projected/7d04cba6-64ad-4a4e-8045-3c91d3237a14-kube-api-access-r9lfh\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:59:08.965024 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.965004    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b46q\" (UniqueName: \"kubernetes.io/projected/92fece83-25d4-4543-8489-67583d71ceff-kube-api-access-2b46q\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:59:08.965479 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.965463    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgtp\" (UniqueName: \"kubernetes.io/projected/017d594a-16ee-456f-8b9f-2fecc89acb9b-kube-api-access-9zgtp\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:59:08.979205 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979183    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-socket-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.979282 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979207    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-slash\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979282 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979223    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-systemd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979282 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979238    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-ovn\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979282 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979257    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-log-socket\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979416 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979301    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-log-socket\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979416 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979302    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-ovn\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979416 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979308    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-slash\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979416 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979310    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-socket-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.979416 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979321    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-dbus\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:59:08.979416 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979342    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-systemd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979416 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979356    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cnibin\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.979416 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979363    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-dbus\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:59:08.979416 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979379    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-cnibin\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.979416 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979389    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.979416 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979415    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-run\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979439    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979468    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-bin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979480    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-run\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979491    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-kubernetes\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979515    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-netns\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979545    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/017d594a-16ee-456f-8b9f-2fecc89acb9b-host\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979545    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-bin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979569    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-netns\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979562    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979578    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-kubernetes\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979582    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/017d594a-16ee-456f-8b9f-2fecc89acb9b-host\") pod \"node-ca-pkxfm\" (UID: \"017d594a-16ee-456f-8b9f-2fecc89acb9b\") " pod="openshift-image-registry/node-ca-pkxfm"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979603    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-bin\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979625    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-multus\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979641    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979667    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-cni-multus\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979671    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-bin\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979697    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92fece83-25d4-4543-8489-67583d71ceff-host-slash\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979715    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-modprobe-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979729    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-device-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979753    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c46b893-7db9-4a89-a701-12fa631394fa-hosts-file\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979795    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c46b893-7db9-4a89-a701-12fa631394fa-hosts-file\") pod \"node-resolver-dxw74\" (UID: \"9c46b893-7db9-4a89-a701-12fa631394fa\") " pod="openshift-dns/node-resolver-dxw74"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979794    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-device-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979808    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979829    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-modprobe-d\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979838    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-system-cni-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979834    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-run-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.979848 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979807    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92fece83-25d4-4543-8489-67583d71ceff-host-slash\") pod \"iptables-alerter-krd42\" (UID: \"92fece83-25d4-4543-8489-67583d71ceff\") " pod="openshift-network-operator/iptables-alerter-krd42"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979862    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-systemd-units\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979898    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysconfig\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979902    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-system-cni-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979901    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-systemd-units\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979926    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-host\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979940    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysconfig\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979951    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-etc-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979967    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-kubelet-config\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979981    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-sys\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979985    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-host\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.979996    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980020    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-var-lib-kubelet\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980035    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-etc-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980060    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-sys\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980067    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-var-lib-kubelet\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980079    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-kubelet\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980098    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cnibin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980078    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cf458915-e617-486b-8b9f-9019d09ca398-kubelet-config\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980126    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-conf-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980152    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-kubelet\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980173    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-cnibin\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980158    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-multus-certs\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980197    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-multus-certs\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980202    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-conf-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980209    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-conf\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980235    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-netd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980251    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-k8s-cni-cncf-io\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980266    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-hostroot\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980280    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980283    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-cni-netd\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980304    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-node-log\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980309    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-k8s-cni-cncf-io\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980307    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-hostroot\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980325    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-system-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980337    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-node-log\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980342    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980345    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-netns\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980376    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-run-netns\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980388    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-system-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980396    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980401    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-sysctl-conf\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980416    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-registration-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980440    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980453    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980463    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-socket-dir-parent\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980486    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-lib-modules\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980493    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-registration-dir\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980500    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-sys-fs\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980512    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-socket-dir-parent\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980516    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-var-lib-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980514    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-cni-dir\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980545    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-var-lib-openvswitch\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980547    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980574    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-sys-fs\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980579    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980583    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-lib-modules\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980580    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980611    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980616    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-kubelet\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980638    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-etc-kubernetes\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980646    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-host-var-lib-kubelet\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980652    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-systemd\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980685    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cea0af3-b692-43ed-8c86-df442fd5378c-etc-systemd\") pod \"tuned-gqgl4\" (UID: \"5cea0af3-b692-43ed-8c86-df442fd5378c\") " pod="openshift-cluster-node-tuning-operator/tuned-gqgl4"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980707    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab43a143-464a-4155-98e7-3f7c480aeca6-crio-socket\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980732    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021ef7d9-672a-4953-bb37-3ca4c271c7ca-etc-kubernetes\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980766    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980802    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab43a143-464a-4155-98e7-3f7c480aeca6-crio-socket\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:08.980814 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:08.980833    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01ab7be9-13bf-4798-bcb0-44ca9923ce0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmqdp\" (UID: \"01ab7be9-13bf-4798-bcb0-44ca9923ce0c\") " pod="openshift-multus/multus-additional-cni-plugins-xmqdp"
Nov 03 06:59:09.022689 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.022624    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 06:59:09.023696 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.023626    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 06:59:09.023838 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.023817    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 06:59:09.024104 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024028    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 06:59:09.024104 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024049    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 06:59:09.024104 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024085    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 06:59:09.024104 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024089    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kl8t2\""
Nov 03 06:59:09.024104 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024106    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 06:59:09.024366 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024128    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 06:59:09.024366 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024175    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 06:59:09.024366 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024208    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 06:59:09.024366 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024259    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 06:59:09.024366 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024287    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 06:59:09.024366 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024309    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 06:59:09.024366 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024288    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 06:59:09.024366 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024327    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44mbd\""
Nov 03 06:59:09.024685 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024424    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kx4sk\""
Nov 03 06:59:09.024685 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024428    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 06:59:09.024685 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024451    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 06:59:09.024685 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024557    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 06:59:09.024685 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024609    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 06:59:09.024685 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024654    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 06:59:09.024685 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024658    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7dznd\""
Nov 03 06:59:09.024685 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024667    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nd4b5\""
Nov 03 06:59:09.024685 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024669    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-br79x\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024704    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-htlhr\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024712    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dftjg\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024722    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8gp29\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024734    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8c889\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024747    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024765    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l9lht\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024776    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dmgt\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024791    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024816    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024894    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5znfn\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.024896    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cb66z\""
Nov 03 06:59:09.025067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.025046    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 06:59:09.025421 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.025121    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovnkube-config\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:09.025421 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.025258    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-cert\") pod \"ingress-canary-6vxzv\" (UID: \"2cee886c-6e9d-431b-9cda-0f8cd618cf2b\") " pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:59:09.025517 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.025504    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 06:59:09.026174 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.026160    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-key\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:59:09.027304 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.027291    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:09.027355 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.027341    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cf458915-e617-486b-8b9f-9019d09ca398-original-pull-secret\") pod \"global-pull-secret-syncer-pw94d\" (UID: \"cf458915-e617-486b-8b9f-9019d09ca398\") " pod="kube-system/global-pull-secret-syncer-pw94d"
Nov 03 06:59:09.027423 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.027408    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f4ea98-f511-4031-8b8b-7d057e1433cd-config-volume\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:09.027575 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.027528    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-service-ca-bundle\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:59:09.027744 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.027730    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-ovn-node-metrics-cert\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:09.027866 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.027852    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03645044-2dfb-4a33-8a26-c3c5684cbe09-signing-cabundle\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:59:09.027984 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.027968    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/021ef7d9-672a-4953-bb37-3ca4c271c7ca-multus-daemon-config\") pod \"multus-2dp6g\" (UID: \"021ef7d9-672a-4953-bb37-3ca4c271c7ca\") " pod="openshift-multus/multus-2dp6g"
Nov 03 06:59:09.028257 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.028237    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-trusted-ca-bundle\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:59:09.029953 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:09.029937    4470 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 06:59:09.030037 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:09.029938    4470 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:09.030037 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:09.030003    4470 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs podName:7d04cba6-64ad-4a4e-8045-3c91d3237a14 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.529988024 +0000 UTC m=+1.862357030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs") pod "network-metrics-daemon-mq4wx" (UID: "7d04cba6-64ad-4a4e-8045-3c91d3237a14") : secret "metrics-daemon-secret" not found
Nov 03 06:59:09.030172 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:09.030044    4470 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.530030758 +0000 UTC m=+1.862399745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:09.030492 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.030473    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b57d\" (UniqueName: \"kubernetes.io/projected/3c56f558-2833-421b-a2a4-fd023736ea7c-kube-api-access-8b57d\") pod \"periodic-gathering-l2dln-4d54l\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") " pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 06:59:09.030569 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:09.030545    4470 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 06:59:09.030631 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:09.030576    4470 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls podName:a5f4ea98-f511-4031-8b8b-7d057e1433cd nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.530568815 +0000 UTC m=+1.862937824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls") pod "dns-default-pcxdc" (UID: "a5f4ea98-f511-4031-8b8b-7d057e1433cd") : secret "dns-default-metrics-tls" not found
Nov 03 06:59:09.031026 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.031004    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljw5\" (UniqueName: \"kubernetes.io/projected/b31cdbad-0b70-43c3-89c8-3fce65ba4ec3-kube-api-access-bljw5\") pod \"aws-ebs-csi-driver-node-hgct7\" (UID: \"b31cdbad-0b70-43c3-89c8-3fce65ba4ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgct7"
Nov 03 06:59:09.031080 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.031024    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v754j\" (UniqueName: \"kubernetes.io/projected/2cee886c-6e9d-431b-9cda-0f8cd618cf2b-kube-api-access-v754j\") pod \"ingress-canary-6vxzv\" (UID: \"2cee886c-6e9d-431b-9cda-0f8cd618cf2b\") " pod="openshift-ingress-canary/ingress-canary-6vxzv"
Nov 03 06:59:09.032467 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.032435    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b44wp\" (UniqueName: \"kubernetes.io/projected/ab43a143-464a-4155-98e7-3f7c480aeca6-kube-api-access-b44wp\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:09.032580 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.032563    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9q6\" (UniqueName: \"kubernetes.io/projected/7bcf499f-e3d7-4146-89f8-74e2ae1685e3-kube-api-access-jn9q6\") pod \"ovnkube-node-6w8lk\" (UID: \"7bcf499f-e3d7-4146-89f8-74e2ae1685e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:09.032613 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.032576    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfk5v\" (UniqueName: \"kubernetes.io/projected/03645044-2dfb-4a33-8a26-c3c5684cbe09-kube-api-access-bfk5v\") pod \"service-ca-5fffc97bdf-6484x\" (UID: \"03645044-2dfb-4a33-8a26-c3c5684cbe09\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6484x"
Nov 03 06:59:09.584258 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.584221    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:59:09.584451 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.584281    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:09.584451 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.584347    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:09.584451 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:09.584353    4470 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 06:59:09.584451 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:09.584437    4470 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs podName:7d04cba6-64ad-4a4e-8045-3c91d3237a14 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:10.584414043 +0000 UTC m=+2.916783027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs") pod "network-metrics-daemon-mq4wx" (UID: "7d04cba6-64ad-4a4e-8045-3c91d3237a14") : secret "metrics-daemon-secret" not found
Nov 03 06:59:09.584592 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:09.584468    4470 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:09.584592 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:09.584535    4470 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:10.584520577 +0000 UTC m=+2.916889563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:09.584691 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.584673    4470 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 06:59:09.586965 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.586945    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5f4ea98-f511-4031-8b8b-7d057e1433cd-metrics-tls\") pod \"dns-default-pcxdc\" (UID: \"a5f4ea98-f511-4031-8b8b-7d057e1433cd\") " pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:09.771584 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.771550    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:09.871491 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:09.868072    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:10.171801 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:10.171730    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcxdc"]
Nov 03 06:59:10.174698 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:10.174669    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f4ea98_f511_4031_8b8b_7d057e1433cd.slice/crio-39e1387009e533a5a2f0fd39d3d741fa2632bafacb2163b89fb1ee4e40826eb1 WatchSource:0}: Error finding container 39e1387009e533a5a2f0fd39d3d741fa2632bafacb2163b89fb1ee4e40826eb1: Status 404 returned error can't find the container with id 39e1387009e533a5a2f0fd39d3d741fa2632bafacb2163b89fb1ee4e40826eb1
Nov 03 06:59:10.176842 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:10.176816    4470 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 06:59:10.284989 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:10.284965    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:10.380025 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:10.379989    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:59:10.593670 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:10.593633    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:10.593856 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:10.593677    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:59:10.593856 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:10.593783    4470 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:10.593856 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:10.593844    4470 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:12.593828403 +0000 UTC m=+4.926197391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:10.593856 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:10.593786    4470 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 06:59:10.594082 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:10.593877    4470 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs podName:7d04cba6-64ad-4a4e-8045-3c91d3237a14 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:12.593870063 +0000 UTC m=+4.926239068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs") pod "network-metrics-daemon-mq4wx" (UID: "7d04cba6-64ad-4a4e-8045-3c91d3237a14") : secret "metrics-daemon-secret" not found
Nov 03 06:59:10.847507 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:10.847431    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcxdc" event={"ID":"a5f4ea98-f511-4031-8b8b-7d057e1433cd","Type":"ContainerStarted","Data":"39e1387009e533a5a2f0fd39d3d741fa2632bafacb2163b89fb1ee4e40826eb1"}
Nov 03 06:59:10.847859 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:10.847558    4470 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 06:59:12.608958 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:12.608920    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:12.608958 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:12.608962    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:59:12.609528 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:12.609073    4470 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:12.609528 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:12.609176    4470 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls podName:ab43a143-464a-4155-98e7-3f7c480aeca6 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:16.609155775 +0000 UTC m=+8.941524747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vprr7" (UID: "ab43a143-464a-4155-98e7-3f7c480aeca6") : secret "insights-runtime-extractor-tls" not found
Nov 03 06:59:12.611399 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:12.611382    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d04cba6-64ad-4a4e-8045-3c91d3237a14-metrics-certs\") pod \"network-metrics-daemon-mq4wx\" (UID: \"7d04cba6-64ad-4a4e-8045-3c91d3237a14\") " pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:59:12.766794 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:12.766773    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mq4wx"
Nov 03 06:59:12.862413 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:12.862332    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcxdc" event={"ID":"a5f4ea98-f511-4031-8b8b-7d057e1433cd","Type":"ContainerStarted","Data":"1978ec6d91e12bb670b279ea34216f8e7fa472a7ba86f694301daa7082e96040"}
Nov 03 06:59:13.170557 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:13.170480    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mq4wx"]
Nov 03 06:59:13.173274 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:13.173246    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d04cba6_64ad_4a4e_8045_3c91d3237a14.slice/crio-7fc512cb35c64f00f670a4b3dc48cbddc3ccaf9705d175bce01fec0e3584013a WatchSource:0}: Error finding container 7fc512cb35c64f00f670a4b3dc48cbddc3ccaf9705d175bce01fec0e3584013a: Status 404 returned error can't find the container with id 7fc512cb35c64f00f670a4b3dc48cbddc3ccaf9705d175bce01fec0e3584013a
Nov 03 06:59:13.866868 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:13.866830    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcxdc" event={"ID":"a5f4ea98-f511-4031-8b8b-7d057e1433cd","Type":"ContainerStarted","Data":"4a9161ec0fc7d388c3c7072cc0b645267b869776833a473db9a82591e754310c"}
Nov 03 06:59:13.867338 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:13.866938    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:13.868108 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:13.868089    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mq4wx" event={"ID":"7d04cba6-64ad-4a4e-8045-3c91d3237a14","Type":"ContainerStarted","Data":"7fc512cb35c64f00f670a4b3dc48cbddc3ccaf9705d175bce01fec0e3584013a"}
Nov 03 06:59:14.877027 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:14.876981    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mq4wx" event={"ID":"7d04cba6-64ad-4a4e-8045-3c91d3237a14","Type":"ContainerStarted","Data":"9b2249af6c57ac84fe40fc5aa4a8b61af11d7a8ca64175aeb6e93b1903fd7332"}
Nov 03 06:59:14.877537 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:14.877069    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mq4wx" event={"ID":"7d04cba6-64ad-4a4e-8045-3c91d3237a14","Type":"ContainerStarted","Data":"638bd1b6c551e49e7080389d17a58fb635d84968bb61da386f9c6177d45e67a9"}
Nov 03 06:59:16.552939 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:16.552908    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6w8lk"
Nov 03 06:59:16.632984 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:16.632947    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:16.635351 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:16.635325    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab43a143-464a-4155-98e7-3f7c480aeca6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vprr7\" (UID: \"ab43a143-464a-4155-98e7-3f7c480aeca6\") " pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:16.670555 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:16.670529    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vprr7"
Nov 03 06:59:17.040906 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:17.040876    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vprr7"]
Nov 03 06:59:17.044279 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:17.044249    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab43a143_464a_4155_98e7_3f7c480aeca6.slice/crio-b28af86537ce566a4eaad8df0ab2fb09a6ae1cd68b4f213803bf93d0853f1d8e WatchSource:0}: Error finding container b28af86537ce566a4eaad8df0ab2fb09a6ae1cd68b4f213803bf93d0853f1d8e: Status 404 returned error can't find the container with id b28af86537ce566a4eaad8df0ab2fb09a6ae1cd68b4f213803bf93d0853f1d8e
Nov 03 06:59:17.885371 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:17.885329    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vprr7" event={"ID":"ab43a143-464a-4155-98e7-3f7c480aeca6","Type":"ContainerStarted","Data":"5f4f4582034be501827a41554aa25a8c5284f160e6e2ffda003cc7e08fedeb3f"}
Nov 03 06:59:17.885725 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:17.885398    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vprr7" event={"ID":"ab43a143-464a-4155-98e7-3f7c480aeca6","Type":"ContainerStarted","Data":"b28af86537ce566a4eaad8df0ab2fb09a6ae1cd68b4f213803bf93d0853f1d8e"}
Nov 03 06:59:18.421609 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:18.421578    4470 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-24.us-west-1.compute.internal" event="NodeReady"
Nov 03 06:59:18.890095 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:18.890062    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vprr7" event={"ID":"ab43a143-464a-4155-98e7-3f7c480aeca6","Type":"ContainerStarted","Data":"3e358882101b8254014f933cb594929953542e79decdc943c0b07aa7dd209938"}
Nov 03 06:59:20.897984 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:20.897949    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vprr7" event={"ID":"ab43a143-464a-4155-98e7-3f7c480aeca6","Type":"ContainerStarted","Data":"6570dca02cdface99321abf617d658e93c669d13c152f647c9cb69332edadc79"}
Nov 03 06:59:21.862985 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.862958    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh"]
Nov 03 06:59:21.864778 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.864746    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh"
Nov 03 06:59:21.913586 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.913561    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-zx2t9"]
Nov 03 06:59:21.915368 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.915351    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-zx2t9"
Nov 03 06:59:21.929378 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.929361    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cnl2g\""
Nov 03 06:59:21.929483 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.929419    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 06:59:21.944099 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.944066    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh"]
Nov 03 06:59:21.972791 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.972765    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/77aba703-0ffc-47c4-b324-c9fd2247c90c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fsmkh\" (UID: \"77aba703-0ffc-47c4-b324-c9fd2247c90c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh"
Nov 03 06:59:21.972895 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.972798    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjbgd\" (UniqueName: \"kubernetes.io/projected/1348f759-d498-4c5e-8fae-6fb2ec68fa59-kube-api-access-vjbgd\") pod \"downloads-6dd789c8b7-zx2t9\" (UID: \"1348f759-d498-4c5e-8fae-6fb2ec68fa59\") " pod="openshift-console/downloads-6dd789c8b7-zx2t9"
Nov 03 06:59:21.972945 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.972929    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1348f759-d498-4c5e-8fae-6fb2ec68fa59-tmp\") pod \"downloads-6dd789c8b7-zx2t9\" (UID: \"1348f759-d498-4c5e-8fae-6fb2ec68fa59\") " pod="openshift-console/downloads-6dd789c8b7-zx2t9"
Nov 03 06:59:21.973881 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.973863    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68b7796766-mrzhx"]
Nov 03 06:59:21.974672 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.974654    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 06:59:21.974742 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.974726    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 06:59:21.975583 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.975567    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-zx2t9"]
Nov 03 06:59:21.975662 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:21.975652    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.025388 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.025358    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-92g55\""
Nov 03 06:59:22.073756 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.073726    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szq6\" (UniqueName: \"kubernetes.io/projected/65bc389e-21c0-46dc-821b-415efef006a3-kube-api-access-4szq6\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.073949 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.073770    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65bc389e-21c0-46dc-821b-415efef006a3-ca-trust-extracted\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.073949 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.073796    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65bc389e-21c0-46dc-821b-415efef006a3-bound-sa-token\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.073949 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.073909    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65bc389e-21c0-46dc-821b-415efef006a3-image-registry-private-configuration\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.073949 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.073947    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65bc389e-21c0-46dc-821b-415efef006a3-trusted-ca\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.074106 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.073978    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65bc389e-21c0-46dc-821b-415efef006a3-registry-certificates\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.074106 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.074067    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1348f759-d498-4c5e-8fae-6fb2ec68fa59-tmp\") pod \"downloads-6dd789c8b7-zx2t9\" (UID: \"1348f759-d498-4c5e-8fae-6fb2ec68fa59\") " pod="openshift-console/downloads-6dd789c8b7-zx2t9"
Nov 03 06:59:22.074224 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.074104    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/77aba703-0ffc-47c4-b324-c9fd2247c90c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fsmkh\" (UID: \"77aba703-0ffc-47c4-b324-c9fd2247c90c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh"
Nov 03 06:59:22.074224 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.074148    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbgd\" (UniqueName: \"kubernetes.io/projected/1348f759-d498-4c5e-8fae-6fb2ec68fa59-kube-api-access-vjbgd\") pod \"downloads-6dd789c8b7-zx2t9\" (UID: \"1348f759-d498-4c5e-8fae-6fb2ec68fa59\") " pod="openshift-console/downloads-6dd789c8b7-zx2t9"
Nov 03 06:59:22.074224 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.074193    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65bc389e-21c0-46dc-821b-415efef006a3-installation-pull-secrets\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.074412 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.074222    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65bc389e-21c0-46dc-821b-415efef006a3-registry-tls\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.074466 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.074433    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1348f759-d498-4c5e-8fae-6fb2ec68fa59-tmp\") pod \"downloads-6dd789c8b7-zx2t9\" (UID: \"1348f759-d498-4c5e-8fae-6fb2ec68fa59\") " pod="openshift-console/downloads-6dd789c8b7-zx2t9"
Nov 03 06:59:22.076618 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.076600    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/77aba703-0ffc-47c4-b324-c9fd2247c90c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fsmkh\" (UID: \"77aba703-0ffc-47c4-b324-c9fd2247c90c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh"
Nov 03 06:59:22.085372 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.083837    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 06:59:22.085372 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.083853    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68b7796766-mrzhx"]
Nov 03 06:59:22.127611 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.127516    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 06:59:22.127611 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.127588    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lnjtr\""
Nov 03 06:59:22.127611 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.127591    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 06:59:22.127789 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.127645    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 06:59:22.140642 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.140617    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbgd\" (UniqueName: \"kubernetes.io/projected/1348f759-d498-4c5e-8fae-6fb2ec68fa59-kube-api-access-vjbgd\") pod \"downloads-6dd789c8b7-zx2t9\" (UID: \"1348f759-d498-4c5e-8fae-6fb2ec68fa59\") " pod="openshift-console/downloads-6dd789c8b7-zx2t9"
Nov 03 06:59:22.174779 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.174754    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh"
Nov 03 06:59:22.174920 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.174899    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65bc389e-21c0-46dc-821b-415efef006a3-ca-trust-extracted\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.174986 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.174935    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65bc389e-21c0-46dc-821b-415efef006a3-bound-sa-token\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.174986 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.174974    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65bc389e-21c0-46dc-821b-415efef006a3-image-registry-private-configuration\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.175084 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.175008    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65bc389e-21c0-46dc-821b-415efef006a3-trusted-ca\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.175084 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.175041    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65bc389e-21c0-46dc-821b-415efef006a3-registry-certificates\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.175220 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.175102    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65bc389e-21c0-46dc-821b-415efef006a3-installation-pull-secrets\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.175220 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.175159    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65bc389e-21c0-46dc-821b-415efef006a3-registry-tls\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.175220 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.175185    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szq6\" (UniqueName: \"kubernetes.io/projected/65bc389e-21c0-46dc-821b-415efef006a3-kube-api-access-4szq6\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.175360 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.175304    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65bc389e-21c0-46dc-821b-415efef006a3-ca-trust-extracted\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.175963 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.175936    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65bc389e-21c0-46dc-821b-415efef006a3-registry-certificates\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.176080 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.176012    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65bc389e-21c0-46dc-821b-415efef006a3-trusted-ca\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.177656 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.177633    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65bc389e-21c0-46dc-821b-415efef006a3-installation-pull-secrets\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.177860 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.177842    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65bc389e-21c0-46dc-821b-415efef006a3-image-registry-private-configuration\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.178065 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.178044    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65bc389e-21c0-46dc-821b-415efef006a3-registry-tls\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.224571 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.224516    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-zx2t9"
Nov 03 06:59:22.244566 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.241560    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szq6\" (UniqueName: \"kubernetes.io/projected/65bc389e-21c0-46dc-821b-415efef006a3-kube-api-access-4szq6\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.244566 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.241780    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65bc389e-21c0-46dc-821b-415efef006a3-bound-sa-token\") pod \"image-registry-68b7796766-mrzhx\" (UID: \"65bc389e-21c0-46dc-821b-415efef006a3\") " pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.286083 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.285839    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.542796 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.542732    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh"]
Nov 03 06:59:22.545399 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:22.545372    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77aba703_0ffc_47c4_b324_c9fd2247c90c.slice/crio-c2dff77e27aad57b06791c5b1c08fa803f9375857f87860273eb5090b8b99018 WatchSource:0}: Error finding container c2dff77e27aad57b06791c5b1c08fa803f9375857f87860273eb5090b8b99018: Status 404 returned error can't find the container with id c2dff77e27aad57b06791c5b1c08fa803f9375857f87860273eb5090b8b99018
Nov 03 06:59:22.601121 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.601100    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-zx2t9"]
Nov 03 06:59:22.603095 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:22.603071    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1348f759_d498_4c5e_8fae_6fb2ec68fa59.slice/crio-91f76bff0db0d26dc79d8b2c8fc1d15462f7c7d76989c54b28e48dc2b4e03bb4 WatchSource:0}: Error finding container 91f76bff0db0d26dc79d8b2c8fc1d15462f7c7d76989c54b28e48dc2b4e03bb4: Status 404 returned error can't find the container with id 91f76bff0db0d26dc79d8b2c8fc1d15462f7c7d76989c54b28e48dc2b4e03bb4
Nov 03 06:59:22.664170 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.664120    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68b7796766-mrzhx"]
Nov 03 06:59:22.667107 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:22.667082    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bc389e_21c0_46dc_821b_415efef006a3.slice/crio-af44dcad5a2c41de8cc3a21ef3cacc63242bff1d54fff6775c099c9c5f8c002b WatchSource:0}: Error finding container af44dcad5a2c41de8cc3a21ef3cacc63242bff1d54fff6775c099c9c5f8c002b: Status 404 returned error can't find the container with id af44dcad5a2c41de8cc3a21ef3cacc63242bff1d54fff6775c099c9c5f8c002b
Nov 03 06:59:22.906454 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.906416    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68b7796766-mrzhx" event={"ID":"65bc389e-21c0-46dc-821b-415efef006a3","Type":"ContainerStarted","Data":"38fbf5073f87295cd9d9d77184072316341930cb9d3240e373b65019febe721b"}
Nov 03 06:59:22.906661 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.906475    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68b7796766-mrzhx" event={"ID":"65bc389e-21c0-46dc-821b-415efef006a3","Type":"ContainerStarted","Data":"af44dcad5a2c41de8cc3a21ef3cacc63242bff1d54fff6775c099c9c5f8c002b"}
Nov 03 06:59:22.906661 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.906501    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:22.907592 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.907568    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-zx2t9" event={"ID":"1348f759-d498-4c5e-8fae-6fb2ec68fa59","Type":"ContainerStarted","Data":"91f76bff0db0d26dc79d8b2c8fc1d15462f7c7d76989c54b28e48dc2b4e03bb4"}
Nov 03 06:59:22.908804 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:22.908783    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh" event={"ID":"77aba703-0ffc-47c4-b324-c9fd2247c90c","Type":"ContainerStarted","Data":"c2dff77e27aad57b06791c5b1c08fa803f9375857f87860273eb5090b8b99018"}
Nov 03 06:59:23.032658 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:23.032604    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68b7796766-mrzhx" podStartSLOduration=2.032589623 podStartE2EDuration="2.032589623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:23.031932466 +0000 UTC m=+15.364301459" watchObservedRunningTime="2026-11-03 06:59:23.032589623 +0000 UTC m=+15.364958609"
Nov 03 06:59:23.604942 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:23.604332    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9v5rw"
Nov 03 06:59:23.879479 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:23.879404    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pcxdc"
Nov 03 06:59:24.919513 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:24.919476    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh" event={"ID":"77aba703-0ffc-47c4-b324-c9fd2247c90c","Type":"ContainerStarted","Data":"4f8cc530c807a075233ac7a1ef95e8d12a4d75848a50a197cd63c330b4161ffe"}
Nov 03 06:59:24.919951 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:24.919751    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh"
Nov 03 06:59:24.925601 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:24.925566    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh"
Nov 03 06:59:25.040545 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:25.040483    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fsmkh" podStartSLOduration=2.523723903 podStartE2EDuration="4.040468139s" totalImagesPullingTime="1.516744236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:25.039839753 +0000 UTC m=+17.372208748" watchObservedRunningTime="2026-11-03 06:59:25.040468139 +0000 UTC m=+17.372837124"
Nov 03 06:59:32.757055 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.757024    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rn5sc"]
Nov 03 06:59:32.759473 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.759446    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.808007 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.807974    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"]
Nov 03 06:59:32.810357 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.810327    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.842474 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.842439    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 06:59:32.842474 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.842471    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 06:59:32.850768 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.850739    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.850936 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.850780    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/785921e2-16fb-4b0a-99b2-228afa4995bd-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.850936 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.850802    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-textfile\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.850936 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.850901    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-tls\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.851102 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.850983    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-wtmp\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.851102 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.851011    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.851102 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.851084    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjqbf\" (UniqueName: \"kubernetes.io/projected/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-api-access-tjqbf\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.851271 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.851113    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65a796fc-52a9-4632-8de1-e68bcc3834b4-metrics-client-ca\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.851271 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.851151    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmc4\" (UniqueName: \"kubernetes.io/projected/65a796fc-52a9-4632-8de1-e68bcc3834b4-kube-api-access-ltmc4\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.851271 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.851179    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.851271 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.851252    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65a796fc-52a9-4632-8de1-e68bcc3834b4-root\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.851468 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.851323    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.851468 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.851350    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/785921e2-16fb-4b0a-99b2-228afa4995bd-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.851468 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.851367    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.851468 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.851383    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65a796fc-52a9-4632-8de1-e68bcc3834b4-sys\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.857667 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.857641    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 06:59:32.857964 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.857939    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 06:59:32.867209 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.867184    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 06:59:32.867323 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.867278    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 06:59:32.867449 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.867433    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-666h2\""
Nov 03 06:59:32.868512 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.868489    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"]
Nov 03 06:59:32.918254 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.918218    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 06:59:32.926761 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.926738    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 06:59:32.926761 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.926744    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 06:59:32.926968 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.926747    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cgml2\""
Nov 03 06:59:32.952714 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952681    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65a796fc-52a9-4632-8de1-e68bcc3834b4-metrics-client-ca\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.952714 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952718    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmc4\" (UniqueName: \"kubernetes.io/projected/65a796fc-52a9-4632-8de1-e68bcc3834b4-kube-api-access-ltmc4\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.952924 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952741    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.952924 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952767    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65a796fc-52a9-4632-8de1-e68bcc3834b4-root\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.952924 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952785    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.952924 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952807    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/785921e2-16fb-4b0a-99b2-228afa4995bd-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.952924 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952832    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.952924 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952851    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65a796fc-52a9-4632-8de1-e68bcc3834b4-sys\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.952924 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952869    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.952924 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952888    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/785921e2-16fb-4b0a-99b2-228afa4995bd-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.952924 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952913    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-textfile\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.953304 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952933    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-tls\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.953304 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952955    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-wtmp\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.953304 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.952970    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.953304 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.953014    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjqbf\" (UniqueName: \"kubernetes.io/projected/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-api-access-tjqbf\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.953304 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.953061    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65a796fc-52a9-4632-8de1-e68bcc3834b4-sys\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.953304 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.953264    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65a796fc-52a9-4632-8de1-e68bcc3834b4-root\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.953606 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.953311    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65a796fc-52a9-4632-8de1-e68bcc3834b4-metrics-client-ca\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.953606 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.953548    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-textfile\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.953760 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.953724    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.953843 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.953814    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/785921e2-16fb-4b0a-99b2-228afa4995bd-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.953957 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.953934    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-wtmp\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.954421 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.954397    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.954586 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.954565    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/785921e2-16fb-4b0a-99b2-228afa4995bd-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.956299 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.956258    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.956395 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.956300    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:32.956873 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.956852    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:32.956922 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:32.956900    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/65a796fc-52a9-4632-8de1-e68bcc3834b4-node-exporter-tls\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:33.023757 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:33.023693    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjqbf\" (UniqueName: \"kubernetes.io/projected/785921e2-16fb-4b0a-99b2-228afa4995bd-kube-api-access-tjqbf\") pod \"kube-state-metrics-5dc8d7d89b-dfv4l\" (UID: \"785921e2-16fb-4b0a-99b2-228afa4995bd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:33.024206 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:33.024186    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmc4\" (UniqueName: \"kubernetes.io/projected/65a796fc-52a9-4632-8de1-e68bcc3834b4-kube-api-access-ltmc4\") pod \"node-exporter-rn5sc\" (UID: \"65a796fc-52a9-4632-8de1-e68bcc3834b4\") " pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:33.073247 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:33.073213    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rn5sc"
Nov 03 06:59:33.084148 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:33.084100    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a796fc_52a9_4632_8de1_e68bcc3834b4.slice/crio-5fc4705a44ead6b2b6b041c453bc445216cb4848b3589c343f312eddb1fe68a1 WatchSource:0}: Error finding container 5fc4705a44ead6b2b6b041c453bc445216cb4848b3589c343f312eddb1fe68a1: Status 404 returned error can't find the container with id 5fc4705a44ead6b2b6b041c453bc445216cb4848b3589c343f312eddb1fe68a1
Nov 03 06:59:33.125232 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:33.125192    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"
Nov 03 06:59:33.550222 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:33.550179    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l"]
Nov 03 06:59:33.553625 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:33.553597    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785921e2_16fb_4b0a_99b2_228afa4995bd.slice/crio-fe90df788a97e88db99178fc4c4a713fd19ae5357d2c29796fb3c88a0d2036ca WatchSource:0}: Error finding container fe90df788a97e88db99178fc4c4a713fd19ae5357d2c29796fb3c88a0d2036ca: Status 404 returned error can't find the container with id fe90df788a97e88db99178fc4c4a713fd19ae5357d2c29796fb3c88a0d2036ca
Nov 03 06:59:33.953188 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:33.953091    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rn5sc" event={"ID":"65a796fc-52a9-4632-8de1-e68bcc3834b4","Type":"ContainerStarted","Data":"5fc4705a44ead6b2b6b041c453bc445216cb4848b3589c343f312eddb1fe68a1"}
Nov 03 06:59:33.954611 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:33.954578    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l" event={"ID":"785921e2-16fb-4b0a-99b2-228afa4995bd","Type":"ContainerStarted","Data":"fe90df788a97e88db99178fc4c4a713fd19ae5357d2c29796fb3c88a0d2036ca"}
Nov 03 06:59:34.765559 ip-10-0-129-24 kubenswrapper[4470]: E1103 06:59:34.765477    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 06:59:34.960993 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:34.960960    4470 generic.go:356] "Generic (PLEG): container finished" podID="65a796fc-52a9-4632-8de1-e68bcc3834b4" containerID="2515c535e2c24452cfa2b403bab475a4acc660f855ed2d3d523a328d966756c5" exitCode=0
Nov 03 06:59:34.961504 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:34.961036    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rn5sc" event={"ID":"65a796fc-52a9-4632-8de1-e68bcc3834b4","Type":"ContainerDied","Data":"2515c535e2c24452cfa2b403bab475a4acc660f855ed2d3d523a328d966756c5"}
Nov 03 06:59:35.420362 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.420328    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d4997f974-cg8jj"]
Nov 03 06:59:35.441241 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.441206    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.488271 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.488236    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d4997f974-cg8jj"]
Nov 03 06:59:35.496736 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.496661    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 06:59:35.496736 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.496664    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 06:59:35.496736 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.496689    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 06:59:35.496987 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.496765    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 06:59:35.496987 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.496813    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-t2sxl\""
Nov 03 06:59:35.496987 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.496898    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 06:59:35.496987 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.496930    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b2fg6u1biim7v\""
Nov 03 06:59:35.573879 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.573842    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-tls\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.573879 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.573886    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/440f457a-b678-4b70-b5cd-07d53dbbef71-tmp\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.574111 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.573905    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46km\" (UniqueName: \"kubernetes.io/projected/440f457a-b678-4b70-b5cd-07d53dbbef71-kube-api-access-q46km\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.574111 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.573970    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.574111 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.574087    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.574265 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.574154    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/440f457a-b678-4b70-b5cd-07d53dbbef71-metrics-client-ca\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.574265 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.574221    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.574265 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.574263    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.574361 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.574286    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-grpc-tls\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.674792 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.674725    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.674792 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.674766    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-grpc-tls\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.674932 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.674799    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-tls\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.674932 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.674824    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/440f457a-b678-4b70-b5cd-07d53dbbef71-tmp\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.674932 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.674841    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46km\" (UniqueName: \"kubernetes.io/projected/440f457a-b678-4b70-b5cd-07d53dbbef71-kube-api-access-q46km\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.674932 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.674862    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.674932 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.674897    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.674932 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.674920    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/440f457a-b678-4b70-b5cd-07d53dbbef71-metrics-client-ca\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.675181 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.674938    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.676193 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.675917    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/440f457a-b678-4b70-b5cd-07d53dbbef71-tmp\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.676451 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.676405    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/440f457a-b678-4b70-b5cd-07d53dbbef71-metrics-client-ca\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.678872 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.678602    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-tls\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.678872 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.678713    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.679871 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.679849    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-grpc-tls\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.679983 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.679953    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.680323 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.680304    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.681398 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.681372    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/440f457a-b678-4b70-b5cd-07d53dbbef71-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.740251 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.740214    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46km\" (UniqueName: \"kubernetes.io/projected/440f457a-b678-4b70-b5cd-07d53dbbef71-kube-api-access-q46km\") pod \"thanos-querier-7d4997f974-cg8jj\" (UID: \"440f457a-b678-4b70-b5cd-07d53dbbef71\") " pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.865098 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.865065    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:35.971165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.970972    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rn5sc" event={"ID":"65a796fc-52a9-4632-8de1-e68bcc3834b4","Type":"ContainerStarted","Data":"d01b1d49d015a9f714d1ba6627fc922b44ab51ce30178153c84a2c97190c3869"}
Nov 03 06:59:35.971165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.971034    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rn5sc" event={"ID":"65a796fc-52a9-4632-8de1-e68bcc3834b4","Type":"ContainerStarted","Data":"8fc8496cd4a0142c2b3b4ec8d73a1284a35034f096da619c9348919900484ccb"}
Nov 03 06:59:35.974017 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.973992    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l" event={"ID":"785921e2-16fb-4b0a-99b2-228afa4995bd","Type":"ContainerStarted","Data":"6e8fa86a786fc6292baa31e822c8ebcb7ed27de98fc8230c7ad4ceb9b64e5681"}
Nov 03 06:59:35.974163 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:35.974046    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l" event={"ID":"785921e2-16fb-4b0a-99b2-228afa4995bd","Type":"ContainerStarted","Data":"a42e7d1e7cad07ee585ec5d10828a88ec9bf7b72e1b0fb7d6223518675090db3"}
Nov 03 06:59:36.099937 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:36.099880    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rn5sc" podStartSLOduration=2.635763849 podStartE2EDuration="4.099862102s" totalImagesPullingTime="1.371716412s" totalInitContainerRuntime="92.381841ms" isStatefulPod=true podCreationTimestamp="2026-11-03 06:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:36.098674666 +0000 UTC m=+28.431043662" watchObservedRunningTime="2026-11-03 06:59:36.099862102 +0000 UTC m=+28.432231097"
Nov 03 06:59:36.314698 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:36.314655    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d4997f974-cg8jj"]
Nov 03 06:59:36.321364 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:36.321323    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440f457a_b678_4b70_b5cd_07d53dbbef71.slice/crio-f91d31dc2a4016ffe09d563b451e669ce666e2b62a40cbe5df542bd2e0b0ee48 WatchSource:0}: Error finding container f91d31dc2a4016ffe09d563b451e669ce666e2b62a40cbe5df542bd2e0b0ee48: Status 404 returned error can't find the container with id f91d31dc2a4016ffe09d563b451e669ce666e2b62a40cbe5df542bd2e0b0ee48
Nov 03 06:59:36.981336 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:36.981280    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l" event={"ID":"785921e2-16fb-4b0a-99b2-228afa4995bd","Type":"ContainerStarted","Data":"15ad2523ec0c849170c47a89f11917bed28e3e6025f47c81f2e3af6fe5355b74"}
Nov 03 06:59:36.983245 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:36.983219    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj" event={"ID":"440f457a-b678-4b70-b5cd-07d53dbbef71","Type":"ContainerStarted","Data":"f91d31dc2a4016ffe09d563b451e669ce666e2b62a40cbe5df542bd2e0b0ee48"}
Nov 03 06:59:37.105807 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.105755    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dfv4l" podStartSLOduration=3.006586244 podStartE2EDuration="5.105739105s" totalImagesPullingTime="2.099152861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:37.105271111 +0000 UTC m=+29.437640106" watchObservedRunningTime="2026-11-03 06:59:37.105739105 +0000 UTC m=+29.438108099"
Nov 03 06:59:37.482602 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.482563    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"]
Nov 03 06:59:37.499128 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.498810    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"
Nov 03 06:59:37.545611 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.545549    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"]
Nov 03 06:59:37.557490 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.557461    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5cssp\""
Nov 03 06:59:37.557696 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.557671    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 06:59:37.591384 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.591344    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be64ab26-6807-4c79-8008-5c565c77a335-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-9hp59\" (UID: \"be64ab26-6807-4c79-8008-5c565c77a335\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"
Nov 03 06:59:37.591562 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.591400    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjnd7\" (UniqueName: \"kubernetes.io/projected/be64ab26-6807-4c79-8008-5c565c77a335-kube-api-access-rjnd7\") pod \"monitoring-plugin-77f4f76594-9hp59\" (UID: \"be64ab26-6807-4c79-8008-5c565c77a335\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"
Nov 03 06:59:37.692077 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.692039    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be64ab26-6807-4c79-8008-5c565c77a335-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-9hp59\" (UID: \"be64ab26-6807-4c79-8008-5c565c77a335\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"
Nov 03 06:59:37.692283 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.692092    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjnd7\" (UniqueName: \"kubernetes.io/projected/be64ab26-6807-4c79-8008-5c565c77a335-kube-api-access-rjnd7\") pod \"monitoring-plugin-77f4f76594-9hp59\" (UID: \"be64ab26-6807-4c79-8008-5c565c77a335\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"
Nov 03 06:59:37.695304 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.695275    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be64ab26-6807-4c79-8008-5c565c77a335-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-9hp59\" (UID: \"be64ab26-6807-4c79-8008-5c565c77a335\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"
Nov 03 06:59:37.758601 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.758536    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjnd7\" (UniqueName: \"kubernetes.io/projected/be64ab26-6807-4c79-8008-5c565c77a335-kube-api-access-rjnd7\") pod \"monitoring-plugin-77f4f76594-9hp59\" (UID: \"be64ab26-6807-4c79-8008-5c565c77a335\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"
Nov 03 06:59:37.827149 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.827108    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"
Nov 03 06:59:37.853073 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.853048    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-55b544b886-7zvwx"]
Nov 03 06:59:37.873210 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.873176    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:37.937910 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.937634    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 06:59:37.937910 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.937703    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6pmeoo45jgvjf\""
Nov 03 06:59:37.937910 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.937751    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 06:59:37.937910 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.937861    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5r2fz\""
Nov 03 06:59:37.937910 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.937862    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 06:59:37.941324 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.938465    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55b544b886-7zvwx"]
Nov 03 06:59:37.942340 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.942320    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 06:59:37.995067 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.995023    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-secret-metrics-server-client-certs\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:37.995546 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.995091    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-metrics-server-audit-profiles\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:37.995546 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.995222    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phttt\" (UniqueName: \"kubernetes.io/projected/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-kube-api-access-phttt\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:37.995546 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.995273    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-audit-log\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:37.995546 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.995302    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-client-ca-bundle\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:37.995546 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.995334    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:37.995546 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:37.995370    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-secret-metrics-server-tls\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.096181 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.096114    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-metrics-server-audit-profiles\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.096355 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.096192    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phttt\" (UniqueName: \"kubernetes.io/projected/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-kube-api-access-phttt\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.096355 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.096228    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-audit-log\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.096355 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.096256    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-client-ca-bundle\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.096355 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.096288    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.096355 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.096336    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-secret-metrics-server-tls\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.096611 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.096474    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-secret-metrics-server-client-certs\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.097307 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.097216    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-metrics-server-audit-profiles\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.097574 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.097516    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-audit-log\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.097878 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.097852    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.099466 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.099423    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-secret-metrics-server-client-certs\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.099576 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.099539    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-secret-metrics-server-tls\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.100819 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.100799    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-client-ca-bundle\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.163701 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.163640    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phttt\" (UniqueName: \"kubernetes.io/projected/1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86-kube-api-access-phttt\") pod \"metrics-server-55b544b886-7zvwx\" (UID: \"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86\") " pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.200046 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.200007    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:38.232414 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.232384    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"]
Nov 03 06:59:38.236473 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:38.236444    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe64ab26_6807_4c79_8008_5c565c77a335.slice/crio-37e0ddc8ac52d6a453c1eb64f88b659ef4c34fa057b2498a3a6b5a40c3b2685e WatchSource:0}: Error finding container 37e0ddc8ac52d6a453c1eb64f88b659ef4c34fa057b2498a3a6b5a40c3b2685e: Status 404 returned error can't find the container with id 37e0ddc8ac52d6a453c1eb64f88b659ef4c34fa057b2498a3a6b5a40c3b2685e
Nov 03 06:59:38.611228 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.610970    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55b544b886-7zvwx"]
Nov 03 06:59:38.614189 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:38.614157    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa8ebc8_e1c5_4fb6_b8fa_02aa5cd45a86.slice/crio-88166fc4acac126a6d9b0e45d78179f3fb73df516beec2b198042c98af992299 WatchSource:0}: Error finding container 88166fc4acac126a6d9b0e45d78179f3fb73df516beec2b198042c98af992299: Status 404 returned error can't find the container with id 88166fc4acac126a6d9b0e45d78179f3fb73df516beec2b198042c98af992299
Nov 03 06:59:38.993066 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.992981    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55b544b886-7zvwx" event={"ID":"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86","Type":"ContainerStarted","Data":"88166fc4acac126a6d9b0e45d78179f3fb73df516beec2b198042c98af992299"}
Nov 03 06:59:38.994338 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:38.994313    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59" event={"ID":"be64ab26-6807-4c79-8008-5c565c77a335","Type":"ContainerStarted","Data":"37e0ddc8ac52d6a453c1eb64f88b659ef4c34fa057b2498a3a6b5a40c3b2685e"}
Nov 03 06:59:40.001917 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:40.001882    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj" event={"ID":"440f457a-b678-4b70-b5cd-07d53dbbef71","Type":"ContainerStarted","Data":"b4aba89da78794997fee04f798b5cfe70b23bcd2af2dc173a54296240cac3d98"}
Nov 03 06:59:40.002417 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:40.001959    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj" event={"ID":"440f457a-b678-4b70-b5cd-07d53dbbef71","Type":"ContainerStarted","Data":"0b61f0dbdbb93f46c9838fc1badaf8e5a6aa51b72728c4fd00ab60bf9a602cb6"}
Nov 03 06:59:40.002417 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:40.001989    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj" event={"ID":"440f457a-b678-4b70-b5cd-07d53dbbef71","Type":"ContainerStarted","Data":"9fdf7921bc99db02f8a7098b4560d02f2a4cdce963f86efa764976d0846a70bc"}
Nov 03 06:59:40.005208 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:40.004052    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-zx2t9" event={"ID":"1348f759-d498-4c5e-8fae-6fb2ec68fa59","Type":"ContainerStarted","Data":"ffb08a2f28d9f693f886eacfe3a3305fdae6c831104142bda4dad91cf18c10f4"}
Nov 03 06:59:40.005208 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:40.005183    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-zx2t9"
Nov 03 06:59:40.006821 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:40.006783    4470 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-zx2t9 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" start-of-body=
Nov 03 06:59:40.006945 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:40.006842    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-zx2t9" podUID="1348f759-d498-4c5e-8fae-6fb2ec68fa59" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused"
Nov 03 06:59:40.131365 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:40.131303    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-zx2t9" podStartSLOduration=1.925062552 podStartE2EDuration="19.131286241s" totalImagesPullingTime="17.206223689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:40.130224727 +0000 UTC m=+32.462593723" watchObservedRunningTime="2026-11-03 06:59:40.131286241 +0000 UTC m=+32.463655237"
Nov 03 06:59:40.383464 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:40.383260    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6bhzs"
Nov 03 06:59:41.011418 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:41.010655    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59" event={"ID":"be64ab26-6807-4c79-8008-5c565c77a335","Type":"ContainerStarted","Data":"013359ba4a17e37378bc4ef1c43bb7211826d9cb790369bf95a4edf78a65ffee"}
Nov 03 06:59:41.012315 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:41.011990    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"
Nov 03 06:59:41.018055 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:41.018029    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59"
Nov 03 06:59:41.022769 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:41.022744    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-zx2t9"
Nov 03 06:59:41.131650 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:41.131547    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9hp59" podStartSLOduration=1.517797367 podStartE2EDuration="4.131530667s" totalImagesPullingTime="2.6137333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:41.130554404 +0000 UTC m=+33.462923405" watchObservedRunningTime="2026-11-03 06:59:41.131530667 +0000 UTC m=+33.463899666"
Nov 03 06:59:42.017334 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.017272    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55b544b886-7zvwx" event={"ID":"1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86","Type":"ContainerStarted","Data":"7dae8ab12b544e07503d2114372b12c37fa96eb5f1eb3639b2d06c021bc33ea0"}
Nov 03 06:59:42.021165 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.021080    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj" event={"ID":"440f457a-b678-4b70-b5cd-07d53dbbef71","Type":"ContainerStarted","Data":"a6579174dd61073c1b69675a4d1aa5e369fae13b91debdde557ea04c1db5f6c3"}
Nov 03 06:59:42.021302 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.021204    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj" event={"ID":"440f457a-b678-4b70-b5cd-07d53dbbef71","Type":"ContainerStarted","Data":"314381a82ab337478c9b70c5ac37a230d9e8d405ba92fc6518a3305a6902d300"}
Nov 03 06:59:42.021302 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.021248    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj" event={"ID":"440f457a-b678-4b70-b5cd-07d53dbbef71","Type":"ContainerStarted","Data":"0435e85a50f69df5ddbbf206ec61b9785b3b6123c9cff83cd24fdf007165a72f"}
Nov 03 06:59:42.143489 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.143418    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-55b544b886-7zvwx" podStartSLOduration=2.224749512 podStartE2EDuration="5.143402255s" totalImagesPullingTime="2.918652743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:42.142842636 +0000 UTC m=+34.475211631" watchObservedRunningTime="2026-11-03 06:59:42.143402255 +0000 UTC m=+34.475771436"
Nov 03 06:59:42.276782 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.276725    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj" podStartSLOduration=2.402553456 podStartE2EDuration="7.276711056s" totalImagesPullingTime="4.8741576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:42.27573871 +0000 UTC m=+34.608107706" watchObservedRunningTime="2026-11-03 06:59:42.276711056 +0000 UTC m=+34.609080050"
Nov 03 06:59:42.541981 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.541901    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 06:59:42.587830 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.587787    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651409    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651468    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651495    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651524    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651555    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651590    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651704    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-web-config\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651755    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651784    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651841    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651883    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.651916    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6vk\" (UniqueName: \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-kube-api-access-gt6vk\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.652055    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.652289    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.652358    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.652456    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.652494    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config-out\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.652530    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.652566    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.652812 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.652599    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.654520 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.654494    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 06:59:42.655849 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.655821    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 06:59:42.656205 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.656172    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 06:59:42.656205 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.656194    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 06:59:42.656350 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.656213    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 06:59:42.656350 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.656224    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 06:59:42.656350 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.656239    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ffrtn\""
Nov 03 06:59:42.656350 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.656253    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 06:59:42.656350 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.656283    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f0melro631hsq\""
Nov 03 06:59:42.656528 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.656355    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 06:59:42.656683 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.656664    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 06:59:42.656683 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.656670    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 06:59:42.701300 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.701266    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 06:59:42.701498 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.701355    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 06:59:42.701498 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.701388    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 06:59:42.753168 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.753122    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-web-config\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.753384 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.753181    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.753384 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.753362    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.753695 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.753542    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.754250 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.754210    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 06:59:42.754364 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.754340    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 06:59:42.754897 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.754522    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.754897 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.754589    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.754897 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.754626    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6vk\" (UniqueName: \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-kube-api-access-gt6vk\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.754897 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.754664    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.754897 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.754764    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.755494 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755434    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.755494 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755478    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.756452 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755516    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config-out\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.756452 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755545    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.756452 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755581    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.756452 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755604    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.756452 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755655    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.756452 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755687    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.756452 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755714    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.756452 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755745    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.756452 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755775    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.756452 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755809    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.758391 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.756894    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.758391 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.756974    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.758391 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.755327    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.761646 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.761626    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config-out\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.762461 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.762441    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.763226 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.762192    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.763354 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.763002    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.763713 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.763690    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.764017 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.763992    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.764094 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.764007    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.764094 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.764057    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-web-config\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.764280 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.764260    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.764636 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.764610    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.764740 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.764670    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.764740 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.764671    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.766936 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.766915    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.767513 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.767491    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.770567 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.770549    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:42.822452 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:42.822421    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6vk\" (UniqueName: \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-kube-api-access-gt6vk\") pod \"prometheus-k8s-0\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:43.025839 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:43.025807    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:43.041121 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:43.041093    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:43.567103 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:43.567065    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 06:59:43.571364 ip-10-0-129-24 kubenswrapper[4470]: W1103 06:59:43.571332    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c1e90c_9c0e_49c3_a67f_7e88a99ab343.slice/crio-6e028bb3e10e6b85c16a7159863e5777e0f3b791a8455a803cf2a03098dd05ff WatchSource:0}: Error finding container 6e028bb3e10e6b85c16a7159863e5777e0f3b791a8455a803cf2a03098dd05ff: Status 404 returned error can't find the container with id 6e028bb3e10e6b85c16a7159863e5777e0f3b791a8455a803cf2a03098dd05ff
Nov 03 06:59:43.918780 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:43.918711    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68b7796766-mrzhx"
Nov 03 06:59:44.030528 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:44.030495    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerStarted","Data":"6e028bb3e10e6b85c16a7159863e5777e0f3b791a8455a803cf2a03098dd05ff"}
Nov 03 06:59:44.037216 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:44.037187    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj"
Nov 03 06:59:46.040966 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:46.040927    4470 generic.go:356] "Generic (PLEG): container finished" podID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerID="e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea" exitCode=0
Nov 03 06:59:46.041632 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:46.041023    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerDied","Data":"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"}
Nov 03 06:59:51.066563 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:51.066524    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerStarted","Data":"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"}
Nov 03 06:59:51.067037 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:51.066635    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerStarted","Data":"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"}
Nov 03 06:59:51.067037 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:51.066683    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerStarted","Data":"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"}
Nov 03 06:59:51.067037 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:51.066726    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerStarted","Data":"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"}
Nov 03 06:59:51.067037 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:51.066762    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerStarted","Data":"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"}
Nov 03 06:59:51.067037 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:51.066802    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerStarted","Data":"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"}
Nov 03 06:59:51.359088 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:51.358996    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.33955008 podStartE2EDuration="9.358983025s" totalImagesPullingTime="5.996320018s" totalInitContainerRuntime="23.112927ms" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:51.307915912 +0000 UTC m=+43.640284907" watchObservedRunningTime="2026-11-03 06:59:51.358983025 +0000 UTC m=+43.691352019"
Nov 03 06:59:53.041956 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:53.041916    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 06:59:58.200126 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:58.200087    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 06:59:58.200773 ip-10-0-129-24 kubenswrapper[4470]: I1103 06:59:58.200176    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 07:00:04.758813 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:00:04.758772    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:00:12.512533 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:12.512495    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-l2dln-4d54l"]
Nov 03 07:00:12.512987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:12.512765    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-l2dln-4d54l" podUID="3c56f558-2833-421b-a2a4-fd023736ea7c" containerName="insights-gathering" containerID="cri-o://8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e" gracePeriod=30
Nov 03 07:00:12.848811 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:12.848779    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 07:00:12.922326 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:12.922290    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c56f558-2833-421b-a2a4-fd023736ea7c-kube-api-access-8b57d\" (UniqueName: \"kubernetes.io/projected/3c56f558-2833-421b-a2a4-fd023736ea7c-kube-api-access-8b57d\") pod \"3c56f558-2833-421b-a2a4-fd023736ea7c\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") "
Nov 03 07:00:12.922326 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:12.922331    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c56f558-2833-421b-a2a4-fd023736ea7c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c56f558-2833-421b-a2a4-fd023736ea7c-archives-path\") pod \"3c56f558-2833-421b-a2a4-fd023736ea7c\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") "
Nov 03 07:00:12.922540 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:12.922365    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-trusted-ca-bundle\") pod \"3c56f558-2833-421b-a2a4-fd023736ea7c\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") "
Nov 03 07:00:12.922540 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:12.922427    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-service-ca-bundle\") pod \"3c56f558-2833-421b-a2a4-fd023736ea7c\" (UID: \"3c56f558-2833-421b-a2a4-fd023736ea7c\") "
Nov 03 07:00:12.922697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:12.922670    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c56f558-2833-421b-a2a4-fd023736ea7c-archives-path" pod "3c56f558-2833-421b-a2a4-fd023736ea7c" (UID: "3c56f558-2833-421b-a2a4-fd023736ea7c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:00:12.922823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:12.922758    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-trusted-ca-bundle" pod "3c56f558-2833-421b-a2a4-fd023736ea7c" (UID: "3c56f558-2833-421b-a2a4-fd023736ea7c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:00:12.922823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:12.922809    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-service-ca-bundle" pod "3c56f558-2833-421b-a2a4-fd023736ea7c" (UID: "3c56f558-2833-421b-a2a4-fd023736ea7c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:00:12.924867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:12.924842    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c56f558-2833-421b-a2a4-fd023736ea7c-kube-api-access-8b57d" pod "3c56f558-2833-421b-a2a4-fd023736ea7c" (UID: "3c56f558-2833-421b-a2a4-fd023736ea7c"). InnerVolumeSpecName "kube-api-access-8b57d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:00:13.023915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.023878    4470 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-service-ca-bundle\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:13.023915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.023908    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b57d\" (UniqueName: \"kubernetes.io/projected/3c56f558-2833-421b-a2a4-fd023736ea7c-kube-api-access-8b57d\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:13.023915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.023919    4470 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c56f558-2833-421b-a2a4-fd023736ea7c-archives-path\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:13.024212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.023929    4470 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c56f558-2833-421b-a2a4-fd023736ea7c-trusted-ca-bundle\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:13.143956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.143868    4470 generic.go:356] "Generic (PLEG): container finished" podID="3c56f558-2833-421b-a2a4-fd023736ea7c" containerID="8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e" exitCode=2
Nov 03 07:00:13.143956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.143914    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l2dln-4d54l"
Nov 03 07:00:13.144168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.143951    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l2dln-4d54l" event={"ID":"3c56f558-2833-421b-a2a4-fd023736ea7c","Type":"ContainerDied","Data":"8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e"}
Nov 03 07:00:13.144168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.144000    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l2dln-4d54l" event={"ID":"3c56f558-2833-421b-a2a4-fd023736ea7c","Type":"ContainerDied","Data":"3c147a484bb1e1327c07a9e5d323904fbcc70a215e17ba690b46198eaf8855b0"}
Nov 03 07:00:13.144168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.144020    4470 scope.go:122] "RemoveContainer" containerID="8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e"
Nov 03 07:00:13.152305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.152291    4470 scope.go:122] "RemoveContainer" containerID="8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e"
Nov 03 07:00:13.152561 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:00:13.152543    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e\": container with ID starting with 8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e not found: ID does not exist" containerID="8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e"
Nov 03 07:00:13.152633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.152571    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e"} err="failed to get container status \"8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e\": rpc error: code = NotFound desc = could not find container \"8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e\": container with ID starting with 8a32d4e5fa38cb3d1c7417a09d080653ab26c1438aa5611b73ee578737c6cd3e not found: ID does not exist"
Nov 03 07:00:13.314633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.314599    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-l2dln-4d54l"]
Nov 03 07:00:13.322365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.322339    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-l2dln-4d54l"]
Nov 03 07:00:13.803242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:13.803212    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c56f558-2833-421b-a2a4-fd023736ea7c" path="/var/lib/kubelet/pods/3c56f558-2833-421b-a2a4-fd023736ea7c/volumes"
Nov 03 07:00:18.205582 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:18.205555    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 07:00:18.209443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:18.209422    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-55b544b886-7zvwx"
Nov 03 07:00:28.807824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:28.807779    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:28.848724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:28.848695    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:29.216173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:29.216147    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:34.758215 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:00:34.758176    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:00:45.177195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.177160    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 07:00:45.177726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.177465    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy-thanos" containerID="cri-o://23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b" gracePeriod=600
Nov 03 07:00:45.177726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.177462    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="prometheus" containerID="cri-o://90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012" gracePeriod=600
Nov 03 07:00:45.177726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.177536    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy-web" containerID="cri-o://d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae" gracePeriod=600
Nov 03 07:00:45.177726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.177558    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="config-reloader" containerID="cri-o://b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424" gracePeriod=600
Nov 03 07:00:45.177726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.177559    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy" containerID="cri-o://784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d" gracePeriod=600
Nov 03 07:00:45.177726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.177549    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="thanos-sidecar" containerID="cri-o://1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748" gracePeriod=600
Nov 03 07:00:45.428841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.428779    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:45.505981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.505951    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-grpc-tls\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.505989    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506007    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506025    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-tls-assets\" (UniqueName: \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-tls-assets\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506047    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-2\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506071    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-db\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506108    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506128    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-serving-certs-ca-bundle\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506184    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-0\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506218    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-kube-api-access-gt6vk\" (UniqueName: \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-kube-api-access-gt6vk\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506243    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-web-config\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-web-config\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506263    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config-out\" (UniqueName: \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config-out\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506296    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-metrics-client-ca\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506364    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-1\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506392    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-kubelet-serving-ca-bundle\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506436    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-trusted-ca-bundle\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506463    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-kube-rbac-proxy\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506490    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-metrics-client-certs\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506482    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-2" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506523    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-tls\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.506617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506556    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-thanos-prometheus-http-client-file\") pod \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\" (UID: \"93c1e90c-9c0e-49c3-a67f-7e88a99ab343\") "
Nov 03 07:00:45.507376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506790    4470 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.507376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.506989    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-serving-certs-ca-bundle" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:00:45.507376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.507308    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-db" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:00:45.508727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.507909    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-kubelet-serving-ca-bundle" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:00:45.508727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.508228    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-trusted-ca-bundle" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:00:45.508727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.508353    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-metrics-client-ca" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:00:45.508727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.508564    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-1" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:00:45.509508 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.509460    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-thanos-prometheus-http-client-file" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:00:45.510080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.510050    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-0" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:00:45.510392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.510365    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-kube-rbac-proxy-web" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:00:45.510473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.510452    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:00:45.510696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.510585    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-thanos-sidecar-tls" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:00:45.510696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.510650    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-grpc-tls" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:00:45.511303 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.511277    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-tls" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:00:45.511583 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.511564    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-kube-rbac-proxy" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:00:45.511647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.511578    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-tls-assets" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:00:45.511647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.511582    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config-out" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:00:45.511742 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.511703    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-kube-api-access-gt6vk" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "kube-api-access-gt6vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:00:45.512123 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.512104    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-metrics-client-certs" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:00:45.521537 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.521515    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-web-config" pod "93c1e90c-9c0e-49c3-a67f-7e88a99ab343" (UID: "93c1e90c-9c0e-49c3-a67f-7e88a99ab343"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:00:45.608200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608163    4470 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608196    4470 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-grpc-tls\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608207    4470 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608216    4470 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608225    4470 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-tls-assets\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608233    4470 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-db\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608241    4470 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608251    4470 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608259    4470 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608267    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6vk\" (UniqueName: \"kubernetes.io/projected/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-kube-api-access-gt6vk\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608275    4470 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-web-config\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608283    4470 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-config-out\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608293    4470 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-metrics-client-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608302    4470 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608310    4470 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608320    4470 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608328    4470 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-kube-rbac-proxy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608337    4470 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-metrics-client-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:45.608426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:45.608345    4470 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93c1e90c-9c0e-49c3-a67f-7e88a99ab343-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:00:46.266392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266358    4470 generic.go:356] "Generic (PLEG): container finished" podID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerID="23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b" exitCode=0
Nov 03 07:00:46.266392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266385    4470 generic.go:356] "Generic (PLEG): container finished" podID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerID="784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d" exitCode=0
Nov 03 07:00:46.266392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266393    4470 generic.go:356] "Generic (PLEG): container finished" podID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerID="d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae" exitCode=0
Nov 03 07:00:46.266392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266398    4470 generic.go:356] "Generic (PLEG): container finished" podID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerID="1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748" exitCode=0
Nov 03 07:00:46.266392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266403    4470 generic.go:356] "Generic (PLEG): container finished" podID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerID="b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424" exitCode=0
Nov 03 07:00:46.266897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266408    4470 generic.go:356] "Generic (PLEG): container finished" podID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerID="90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012" exitCode=0
Nov 03 07:00:46.266897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266435    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.266897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266437    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerDied","Data":"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"}
Nov 03 07:00:46.266897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266508    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerDied","Data":"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"}
Nov 03 07:00:46.266897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266534    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerDied","Data":"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"}
Nov 03 07:00:46.266897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266557    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerDied","Data":"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"}
Nov 03 07:00:46.266897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266580    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerDied","Data":"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"}
Nov 03 07:00:46.266897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266580    4470 scope.go:122] "RemoveContainer" containerID="23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"
Nov 03 07:00:46.266897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266602    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerDied","Data":"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"}
Nov 03 07:00:46.266897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.266624    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c1e90c-9c0e-49c3-a67f-7e88a99ab343","Type":"ContainerDied","Data":"6e028bb3e10e6b85c16a7159863e5777e0f3b791a8455a803cf2a03098dd05ff"}
Nov 03 07:00:46.274299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.274283    4470 scope.go:122] "RemoveContainer" containerID="784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"
Nov 03 07:00:46.281562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.281542    4470 scope.go:122] "RemoveContainer" containerID="d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"
Nov 03 07:00:46.289323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.289303    4470 scope.go:122] "RemoveContainer" containerID="1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"
Nov 03 07:00:46.297530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.297512    4470 scope.go:122] "RemoveContainer" containerID="b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"
Nov 03 07:00:46.303952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.303936    4470 scope.go:122] "RemoveContainer" containerID="90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"
Nov 03 07:00:46.315788 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.315761    4470 scope.go:122] "RemoveContainer" containerID="e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"
Nov 03 07:00:46.322543 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.322520    4470 scope.go:122] "RemoveContainer" containerID="23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"
Nov 03 07:00:46.322833 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:00:46.322811    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": container with ID starting with 23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b not found: ID does not exist" containerID="23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"
Nov 03 07:00:46.322897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.322845    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"} err="failed to get container status \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": rpc error: code = NotFound desc = could not find container \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": container with ID starting with 23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b not found: ID does not exist"
Nov 03 07:00:46.322897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.322866    4470 scope.go:122] "RemoveContainer" containerID="784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"
Nov 03 07:00:46.323100 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:00:46.323083    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": container with ID starting with 784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d not found: ID does not exist" containerID="784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"
Nov 03 07:00:46.323158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.323107    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"} err="failed to get container status \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": rpc error: code = NotFound desc = could not find container \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": container with ID starting with 784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d not found: ID does not exist"
Nov 03 07:00:46.323158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.323148    4470 scope.go:122] "RemoveContainer" containerID="d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"
Nov 03 07:00:46.323353 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:00:46.323338    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": container with ID starting with d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae not found: ID does not exist" containerID="d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"
Nov 03 07:00:46.323390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.323357    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"} err="failed to get container status \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": rpc error: code = NotFound desc = could not find container \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": container with ID starting with d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae not found: ID does not exist"
Nov 03 07:00:46.323390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.323369    4470 scope.go:122] "RemoveContainer" containerID="1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"
Nov 03 07:00:46.323610 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:00:46.323582    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": container with ID starting with 1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748 not found: ID does not exist" containerID="1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"
Nov 03 07:00:46.323652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.323616    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"} err="failed to get container status \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": rpc error: code = NotFound desc = could not find container \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": container with ID starting with 1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748 not found: ID does not exist"
Nov 03 07:00:46.323652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.323634    4470 scope.go:122] "RemoveContainer" containerID="b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"
Nov 03 07:00:46.323917 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:00:46.323902    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": container with ID starting with b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424 not found: ID does not exist" containerID="b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"
Nov 03 07:00:46.323963 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.323922    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"} err="failed to get container status \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": rpc error: code = NotFound desc = could not find container \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": container with ID starting with b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424 not found: ID does not exist"
Nov 03 07:00:46.323963 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.323939    4470 scope.go:122] "RemoveContainer" containerID="90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"
Nov 03 07:00:46.324155 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:00:46.324124    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": container with ID starting with 90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012 not found: ID does not exist" containerID="90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"
Nov 03 07:00:46.324197 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.324158    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"} err="failed to get container status \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": rpc error: code = NotFound desc = could not find container \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": container with ID starting with 90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012 not found: ID does not exist"
Nov 03 07:00:46.324197 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.324171    4470 scope.go:122] "RemoveContainer" containerID="e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"
Nov 03 07:00:46.324387 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:00:46.324374    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": container with ID starting with e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea not found: ID does not exist" containerID="e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"
Nov 03 07:00:46.324431 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.324390    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"} err="failed to get container status \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": rpc error: code = NotFound desc = could not find container \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": container with ID starting with e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea not found: ID does not exist"
Nov 03 07:00:46.324431 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.324401    4470 scope.go:122] "RemoveContainer" containerID="23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"
Nov 03 07:00:46.324576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.324558    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"} err="failed to get container status \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": rpc error: code = NotFound desc = could not find container \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": container with ID starting with 23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b not found: ID does not exist"
Nov 03 07:00:46.324623 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.324576    4470 scope.go:122] "RemoveContainer" containerID="784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"
Nov 03 07:00:46.324775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.324753    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"} err="failed to get container status \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": rpc error: code = NotFound desc = could not find container \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": container with ID starting with 784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d not found: ID does not exist"
Nov 03 07:00:46.324822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.324776    4470 scope.go:122] "RemoveContainer" containerID="d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"
Nov 03 07:00:46.324957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.324940    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"} err="failed to get container status \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": rpc error: code = NotFound desc = could not find container \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": container with ID starting with d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae not found: ID does not exist"
Nov 03 07:00:46.324993 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.324957    4470 scope.go:122] "RemoveContainer" containerID="1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"
Nov 03 07:00:46.325239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.325181    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"} err="failed to get container status \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": rpc error: code = NotFound desc = could not find container \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": container with ID starting with 1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748 not found: ID does not exist"
Nov 03 07:00:46.325239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.325205    4470 scope.go:122] "RemoveContainer" containerID="b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"
Nov 03 07:00:46.325413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.325394    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"} err="failed to get container status \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": rpc error: code = NotFound desc = could not find container \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": container with ID starting with b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424 not found: ID does not exist"
Nov 03 07:00:46.325456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.325415    4470 scope.go:122] "RemoveContainer" containerID="90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"
Nov 03 07:00:46.325594 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.325577    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"} err="failed to get container status \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": rpc error: code = NotFound desc = could not find container \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": container with ID starting with 90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012 not found: ID does not exist"
Nov 03 07:00:46.325631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.325594    4470 scope.go:122] "RemoveContainer" containerID="e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"
Nov 03 07:00:46.325793 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.325778    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"} err="failed to get container status \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": rpc error: code = NotFound desc = could not find container \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": container with ID starting with e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea not found: ID does not exist"
Nov 03 07:00:46.325834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.325794    4470 scope.go:122] "RemoveContainer" containerID="23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"
Nov 03 07:00:46.326004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.325986    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"} err="failed to get container status \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": rpc error: code = NotFound desc = could not find container \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": container with ID starting with 23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b not found: ID does not exist"
Nov 03 07:00:46.326071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.326005    4470 scope.go:122] "RemoveContainer" containerID="784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"
Nov 03 07:00:46.326247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.326231    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"} err="failed to get container status \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": rpc error: code = NotFound desc = could not find container \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": container with ID starting with 784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d not found: ID does not exist"
Nov 03 07:00:46.326297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.326248    4470 scope.go:122] "RemoveContainer" containerID="d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"
Nov 03 07:00:46.326453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.326437    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"} err="failed to get container status \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": rpc error: code = NotFound desc = could not find container \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": container with ID starting with d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae not found: ID does not exist"
Nov 03 07:00:46.326453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.326452    4470 scope.go:122] "RemoveContainer" containerID="1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"
Nov 03 07:00:46.326640 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.326623    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"} err="failed to get container status \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": rpc error: code = NotFound desc = could not find container \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": container with ID starting with 1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748 not found: ID does not exist"
Nov 03 07:00:46.326705 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.326641    4470 scope.go:122] "RemoveContainer" containerID="b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"
Nov 03 07:00:46.326838 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.326819    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"} err="failed to get container status \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": rpc error: code = NotFound desc = could not find container \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": container with ID starting with b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424 not found: ID does not exist"
Nov 03 07:00:46.326882 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.326838    4470 scope.go:122] "RemoveContainer" containerID="90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"
Nov 03 07:00:46.327024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327007    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"} err="failed to get container status \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": rpc error: code = NotFound desc = could not find container \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": container with ID starting with 90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012 not found: ID does not exist"
Nov 03 07:00:46.327024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327023    4470 scope.go:122] "RemoveContainer" containerID="e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"
Nov 03 07:00:46.327212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327195    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"} err="failed to get container status \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": rpc error: code = NotFound desc = could not find container \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": container with ID starting with e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea not found: ID does not exist"
Nov 03 07:00:46.327261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327211    4470 scope.go:122] "RemoveContainer" containerID="23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"
Nov 03 07:00:46.327391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327373    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"} err="failed to get container status \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": rpc error: code = NotFound desc = could not find container \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": container with ID starting with 23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b not found: ID does not exist"
Nov 03 07:00:46.327391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327390    4470 scope.go:122] "RemoveContainer" containerID="784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"
Nov 03 07:00:46.327568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327552    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"} err="failed to get container status \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": rpc error: code = NotFound desc = could not find container \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": container with ID starting with 784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d not found: ID does not exist"
Nov 03 07:00:46.327608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327568    4470 scope.go:122] "RemoveContainer" containerID="d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"
Nov 03 07:00:46.327766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327751    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"} err="failed to get container status \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": rpc error: code = NotFound desc = could not find container \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": container with ID starting with d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae not found: ID does not exist"
Nov 03 07:00:46.327812 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327766    4470 scope.go:122] "RemoveContainer" containerID="1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"
Nov 03 07:00:46.327947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327932    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"} err="failed to get container status \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": rpc error: code = NotFound desc = could not find container \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": container with ID starting with 1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748 not found: ID does not exist"
Nov 03 07:00:46.327985 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.327947    4470 scope.go:122] "RemoveContainer" containerID="b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"
Nov 03 07:00:46.328163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.328126    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"} err="failed to get container status \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": rpc error: code = NotFound desc = could not find container \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": container with ID starting with b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424 not found: ID does not exist"
Nov 03 07:00:46.328205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.328165    4470 scope.go:122] "RemoveContainer" containerID="90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"
Nov 03 07:00:46.328365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.328351    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"} err="failed to get container status \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": rpc error: code = NotFound desc = could not find container \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": container with ID starting with 90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012 not found: ID does not exist"
Nov 03 07:00:46.328410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.328365    4470 scope.go:122] "RemoveContainer" containerID="e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"
Nov 03 07:00:46.328559 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.328541    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"} err="failed to get container status \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": rpc error: code = NotFound desc = could not find container \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": container with ID starting with e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea not found: ID does not exist"
Nov 03 07:00:46.328602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.328560    4470 scope.go:122] "RemoveContainer" containerID="23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"
Nov 03 07:00:46.328736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.328721    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"} err="failed to get container status \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": rpc error: code = NotFound desc = could not find container \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": container with ID starting with 23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b not found: ID does not exist"
Nov 03 07:00:46.328736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.328735    4470 scope.go:122] "RemoveContainer" containerID="784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"
Nov 03 07:00:46.328901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.328886    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"} err="failed to get container status \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": rpc error: code = NotFound desc = could not find container \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": container with ID starting with 784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d not found: ID does not exist"
Nov 03 07:00:46.328938 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.328901    4470 scope.go:122] "RemoveContainer" containerID="d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"
Nov 03 07:00:46.329091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.329075    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"} err="failed to get container status \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": rpc error: code = NotFound desc = could not find container \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": container with ID starting with d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae not found: ID does not exist"
Nov 03 07:00:46.329164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.329092    4470 scope.go:122] "RemoveContainer" containerID="1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"
Nov 03 07:00:46.329317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.329302    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"} err="failed to get container status \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": rpc error: code = NotFound desc = could not find container \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": container with ID starting with 1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748 not found: ID does not exist"
Nov 03 07:00:46.329317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.329316    4470 scope.go:122] "RemoveContainer" containerID="b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"
Nov 03 07:00:46.329507 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.329492    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"} err="failed to get container status \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": rpc error: code = NotFound desc = could not find container \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": container with ID starting with b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424 not found: ID does not exist"
Nov 03 07:00:46.329546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.329506    4470 scope.go:122] "RemoveContainer" containerID="90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"
Nov 03 07:00:46.329710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.329693    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"} err="failed to get container status \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": rpc error: code = NotFound desc = could not find container \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": container with ID starting with 90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012 not found: ID does not exist"
Nov 03 07:00:46.329755 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.329712    4470 scope.go:122] "RemoveContainer" containerID="e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"
Nov 03 07:00:46.329883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.329869    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"} err="failed to get container status \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": rpc error: code = NotFound desc = could not find container \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": container with ID starting with e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea not found: ID does not exist"
Nov 03 07:00:46.329883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.329882    4470 scope.go:122] "RemoveContainer" containerID="23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"
Nov 03 07:00:46.330048 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330033    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b"} err="failed to get container status \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": rpc error: code = NotFound desc = could not find container \"23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b\": container with ID starting with 23fa42d61e47ab4acc092e00351fb9b9593e36a61feee2d2526f6ba9a1d4256b not found: ID does not exist"
Nov 03 07:00:46.330087 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330048    4470 scope.go:122] "RemoveContainer" containerID="784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"
Nov 03 07:00:46.330228 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330213    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d"} err="failed to get container status \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": rpc error: code = NotFound desc = could not find container \"784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d\": container with ID starting with 784d6f1a1486c709c2a945c12dcc53611eda3ae14858c7e81eb31f3d54567a9d not found: ID does not exist"
Nov 03 07:00:46.330280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330228    4470 scope.go:122] "RemoveContainer" containerID="d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"
Nov 03 07:00:46.330396 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330382    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae"} err="failed to get container status \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": rpc error: code = NotFound desc = could not find container \"d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae\": container with ID starting with d5288231f371ea56bbe566d6b173ff1757214aca27a2de2f1f532db6eeed65ae not found: ID does not exist"
Nov 03 07:00:46.330442 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330396    4470 scope.go:122] "RemoveContainer" containerID="1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"
Nov 03 07:00:46.330573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330557    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748"} err="failed to get container status \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": rpc error: code = NotFound desc = could not find container \"1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748\": container with ID starting with 1656c1bf6ab6f6dd8672bab90e96ccc41deda74c687913889adca282003da748 not found: ID does not exist"
Nov 03 07:00:46.330616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330573    4470 scope.go:122] "RemoveContainer" containerID="b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"
Nov 03 07:00:46.330761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330746    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424"} err="failed to get container status \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": rpc error: code = NotFound desc = could not find container \"b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424\": container with ID starting with b0af91d1658da703496ba23c029ecb8ca8cc7a6cdbe144a3efc4433c1ace0424 not found: ID does not exist"
Nov 03 07:00:46.330802 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330760    4470 scope.go:122] "RemoveContainer" containerID="90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"
Nov 03 07:00:46.330956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330940    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012"} err="failed to get container status \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": rpc error: code = NotFound desc = could not find container \"90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012\": container with ID starting with 90b271b6dbec2267703cfe39043fc82e9cd21bb0a95a92a4bbb613b789a87012 not found: ID does not exist"
Nov 03 07:00:46.331001 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.330956    4470 scope.go:122] "RemoveContainer" containerID="e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"
Nov 03 07:00:46.331165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.331150    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea"} err="failed to get container status \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": rpc error: code = NotFound desc = could not find container \"e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea\": container with ID starting with e342ab8c62981da3b769f96900490c2ea6f708e32a996cefe7cca23f6bc78bea not found: ID does not exist"
Nov 03 07:00:46.493994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.493915    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 07:00:46.544698 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.544663    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 07:00:46.545428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545408    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 07:00:46.545692 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545680    4470 cpu_manager.go:414] "containerMap: removing container" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="prometheus"
Nov 03 07:00:46.545744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545733    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="prometheus"
Nov 03 07:00:46.545777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545745    4470 cpu_manager.go:414] "containerMap: removing container" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy"
Nov 03 07:00:46.545777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545751    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy"
Nov 03 07:00:46.545777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545760    4470 cpu_manager.go:414] "containerMap: removing container" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy-web"
Nov 03 07:00:46.545777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545765    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy-web"
Nov 03 07:00:46.545919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545779    4470 cpu_manager.go:414] "containerMap: removing container" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="thanos-sidecar"
Nov 03 07:00:46.545919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545784    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="thanos-sidecar"
Nov 03 07:00:46.545919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545794    4470 cpu_manager.go:414] "containerMap: removing container" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy-thanos"
Nov 03 07:00:46.545919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545801    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy-thanos"
Nov 03 07:00:46.545919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545813    4470 cpu_manager.go:414] "containerMap: removing container" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="init-config-reloader"
Nov 03 07:00:46.545919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545819    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="init-config-reloader"
Nov 03 07:00:46.545919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545829    4470 cpu_manager.go:414] "containerMap: removing container" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="config-reloader"
Nov 03 07:00:46.545919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545837    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="config-reloader"
Nov 03 07:00:46.545919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545851    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3c56f558-2833-421b-a2a4-fd023736ea7c" containerName="insights-gathering"
Nov 03 07:00:46.545919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545857    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c56f558-2833-421b-a2a4-fd023736ea7c" containerName="insights-gathering"
Nov 03 07:00:46.546246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545926    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="thanos-sidecar"
Nov 03 07:00:46.546246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545936    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy-thanos"
Nov 03 07:00:46.546246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545943    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="config-reloader"
Nov 03 07:00:46.546246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545948    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c56f558-2833-421b-a2a4-fd023736ea7c" containerName="insights-gathering"
Nov 03 07:00:46.546246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545955    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="prometheus"
Nov 03 07:00:46.546246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545961    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy"
Nov 03 07:00:46.546246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.545966    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" containerName="kube-rbac-proxy-web"
Nov 03 07:00:46.549641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.549627    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.605676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.605648    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 07:00:46.605676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.605678    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f0melro631hsq\""
Nov 03 07:00:46.605982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.605959    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 07:00:46.605982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.605979    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 07:00:46.606080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.605984    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 07:00:46.606080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.605964    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 07:00:46.606080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.606046    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 07:00:46.606207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.606178    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 07:00:46.606257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.606223    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ffrtn\""
Nov 03 07:00:46.606257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.606236    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 07:00:46.606338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.606257    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 07:00:46.606338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.606269    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 07:00:46.606397 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.606360    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 07:00:46.606582 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.606569    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 07:00:46.657798 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.657774    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 07:00:46.657956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.657818    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 07:00:46.716633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.716603    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-web-config\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.716788 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.716639    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.716788 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.716667    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.716860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.716781    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.716860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.716813    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.716860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.716836    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.716860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.716850    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.716978 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.716868    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.716978 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.716929    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-config\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.716978 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.716946    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/982afad1-febf-46f3-b82d-ce986908849e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.716978 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.716972    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.717088 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.717004    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.717088 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.717033    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.717165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.717092    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.717165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.717124    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/982afad1-febf-46f3-b82d-ce986908849e-config-out\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.717229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.717171    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.717229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.717204    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mprsc\" (UniqueName: \"kubernetes.io/projected/982afad1-febf-46f3-b82d-ce986908849e-kube-api-access-mprsc\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.717229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.717225    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.717314 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.717294    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.717348 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.717310    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.788094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.788003    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 07:00:46.818439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818405    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/982afad1-febf-46f3-b82d-ce986908849e-config-out\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.818439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818440    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.818672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818458    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mprsc\" (UniqueName: \"kubernetes.io/projected/982afad1-febf-46f3-b82d-ce986908849e-kube-api-access-mprsc\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.818672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818474    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.818672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818500    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.818672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818514    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.818672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818551    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-web-config\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.818672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818566    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.818672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818585    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.818672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818609    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.818672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818632    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.818672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818664    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818688    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818716    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818744    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-config\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818763    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/982afad1-febf-46f3-b82d-ce986908849e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818788    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818807    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818830    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.818864    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.819220    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.819277    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.819849    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.819898    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.820113    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.820488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.820201    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.822574 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.822550    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.822903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.822707    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.823233 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.823207    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.823334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.823209    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.824078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.823880    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-web-config\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.824206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.824088    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.824290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.824244    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/982afad1-febf-46f3-b82d-ce986908849e-config-out\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.824290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.824280    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.824697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.824500    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/982afad1-febf-46f3-b82d-ce986908849e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.824781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.824736    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/982afad1-febf-46f3-b82d-ce986908849e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.825105 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.825073    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.825470 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.825432    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-config\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.825598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.825580    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/982afad1-febf-46f3-b82d-ce986908849e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:46.883543 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:46.883513    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mprsc\" (UniqueName: \"kubernetes.io/projected/982afad1-febf-46f3-b82d-ce986908849e-kube-api-access-mprsc\") pod \"prometheus-k8s-0\" (UID: \"982afad1-febf-46f3-b82d-ce986908849e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:47.166207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:47.166165    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:00:47.692122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:47.692090    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 07:00:47.752914 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:00:47.752885    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982afad1_febf_46f3_b82d_ce986908849e.slice/crio-78ebbfa23ccd21eaaa619fab5e5350ae25732f7abbde4566701f07607b5144cf WatchSource:0}: Error finding container 78ebbfa23ccd21eaaa619fab5e5350ae25732f7abbde4566701f07607b5144cf: Status 404 returned error can't find the container with id 78ebbfa23ccd21eaaa619fab5e5350ae25732f7abbde4566701f07607b5144cf
Nov 03 07:00:47.803933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:47.803907    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c1e90c-9c0e-49c3-a67f-7e88a99ab343" path="/var/lib/kubelet/pods/93c1e90c-9c0e-49c3-a67f-7e88a99ab343/volumes"
Nov 03 07:00:48.275997 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:48.275961    4470 generic.go:356] "Generic (PLEG): container finished" podID="982afad1-febf-46f3-b82d-ce986908849e" containerID="c3778d67abdac1dbd73dd833c26a07a4b6bd57f0ffe671f7cd1b07c182b7664b" exitCode=0
Nov 03 07:00:48.275997 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:48.275999    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"982afad1-febf-46f3-b82d-ce986908849e","Type":"ContainerDied","Data":"c3778d67abdac1dbd73dd833c26a07a4b6bd57f0ffe671f7cd1b07c182b7664b"}
Nov 03 07:00:48.276232 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:48.276074    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"982afad1-febf-46f3-b82d-ce986908849e","Type":"ContainerStarted","Data":"78ebbfa23ccd21eaaa619fab5e5350ae25732f7abbde4566701f07607b5144cf"}
Nov 03 07:00:49.283510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:49.283474    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"982afad1-febf-46f3-b82d-ce986908849e","Type":"ContainerStarted","Data":"fa45119d0a1ffae7e15f457c79ee5183aefba78ccef497ce4ffa9bc9430cc7f5"}
Nov 03 07:00:49.283891 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:49.283543    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"982afad1-febf-46f3-b82d-ce986908849e","Type":"ContainerStarted","Data":"210c5bcc2a5939c7d2215cd37409f67456694aa2f3ac938212fd9cb7c23920e5"}
Nov 03 07:00:49.283891 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:49.283574    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"982afad1-febf-46f3-b82d-ce986908849e","Type":"ContainerStarted","Data":"a2b761d9d61b69003013ce072bdc558c642218df5829af5dac5bbd8961b8a37b"}
Nov 03 07:00:49.283891 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:49.283601    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"982afad1-febf-46f3-b82d-ce986908849e","Type":"ContainerStarted","Data":"e8e5ab31ad9904fd53b0354fdd71461d5e83dfb703dfe5fdf446514c17d788fb"}
Nov 03 07:00:49.283891 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:49.283630    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"982afad1-febf-46f3-b82d-ce986908849e","Type":"ContainerStarted","Data":"cfdc01a45924762ea8f80c0d2f528ac91aa1c431d2d03ba0ecdf54fc0d51453f"}
Nov 03 07:00:49.283891 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:49.283657    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"982afad1-febf-46f3-b82d-ce986908849e","Type":"ContainerStarted","Data":"fddea3b6edddcbb43dbe0977d11cfa4e16e9a24e5f532781934171ccfe337c68"}
Nov 03 07:00:49.422074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:49.422011    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.406087403 podStartE2EDuration="3.421991388s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.903985ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:00:49.420555835 +0000 UTC m=+101.752924828" watchObservedRunningTime="2026-11-03 07:00:49.421991388 +0000 UTC m=+101.754360383"
Nov 03 07:00:52.166667 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:00:52.166626    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:01:04.758647 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:01:04.758606    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:01:34.757626 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:01:34.757534    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:01:47.167072 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:01:47.167032    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:01:47.183367 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:01:47.183341    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:01:47.514207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:01:47.514121    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 07:02:04.757692 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:02:04.757656    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:02:34.758349 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:02:34.758314    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:03:04.757940 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:03:04.757905    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:03:34.758278 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:03:34.758237    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:04:04.758377 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:04:04.758342    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:04:34.758035 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:04:34.757947    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:05:04.757866 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:05:04.757824    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:05:29.456641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.456607    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"]
Nov 03 07:05:29.459982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.459962    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.517742 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.517707    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 07:05:29.517742 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.517706    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 07:05:29.517944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.517755    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 07:05:29.529330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.529304    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkc55\" (UniqueName: \"kubernetes.io/projected/97744a3d-4e77-4c40-a667-2c4e71576378-kube-api-access-tkc55\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.529391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.529339    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97744a3d-4e77-4c40-a667-2c4e71576378-manifests\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.529391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.529363    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97744a3d-4e77-4c40-a667-2c4e71576378-webhook-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.529528 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.529506    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97744a3d-4e77-4c40-a667-2c4e71576378-tmp\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.529591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.529576    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97744a3d-4e77-4c40-a667-2c4e71576378-metrics-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.529645 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.529625    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97744a3d-4e77-4c40-a667-2c4e71576378-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.567974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.567934    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 07:05:29.568182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.568025    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-frrnv\""
Nov 03 07:05:29.568182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.568093    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 07:05:29.569644 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.569625    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"]
Nov 03 07:05:29.630220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.630193    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97744a3d-4e77-4c40-a667-2c4e71576378-metrics-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.630220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.630225    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97744a3d-4e77-4c40-a667-2c4e71576378-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.630474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.630274    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkc55\" (UniqueName: \"kubernetes.io/projected/97744a3d-4e77-4c40-a667-2c4e71576378-kube-api-access-tkc55\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.630474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.630293    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97744a3d-4e77-4c40-a667-2c4e71576378-manifests\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.630474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.630308    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97744a3d-4e77-4c40-a667-2c4e71576378-webhook-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.630474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.630330    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97744a3d-4e77-4c40-a667-2c4e71576378-tmp\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.630825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.630792    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97744a3d-4e77-4c40-a667-2c4e71576378-manifests\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.632707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.632678    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97744a3d-4e77-4c40-a667-2c4e71576378-tmp\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.632978 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.632958    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97744a3d-4e77-4c40-a667-2c4e71576378-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.633073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.633050    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97744a3d-4e77-4c40-a667-2c4e71576378-webhook-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.633155 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.633081    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97744a3d-4e77-4c40-a667-2c4e71576378-metrics-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.698423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.698390    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkc55\" (UniqueName: \"kubernetes.io/projected/97744a3d-4e77-4c40-a667-2c4e71576378-kube-api-access-tkc55\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm\" (UID: \"97744a3d-4e77-4c40-a667-2c4e71576378\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:29.775934 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:29.775853    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:30.362257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:30.362225    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"]
Nov 03 07:05:30.414560 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:05:30.414529    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97744a3d_4e77_4c40_a667_2c4e71576378.slice/crio-c805d7923daf29948ee68a1f17d8c1dd8cd7729492942a66be58bbbcacd4f9d5 WatchSource:0}: Error finding container c805d7923daf29948ee68a1f17d8c1dd8cd7729492942a66be58bbbcacd4f9d5: Status 404 returned error can't find the container with id c805d7923daf29948ee68a1f17d8c1dd8cd7729492942a66be58bbbcacd4f9d5
Nov 03 07:05:30.416085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:30.416069    4470 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 07:05:31.272346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:31.272308    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm" event={"ID":"97744a3d-4e77-4c40-a667-2c4e71576378","Type":"ContainerStarted","Data":"c805d7923daf29948ee68a1f17d8c1dd8cd7729492942a66be58bbbcacd4f9d5"}
Nov 03 07:05:34.285580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:34.285500    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm" event={"ID":"97744a3d-4e77-4c40-a667-2c4e71576378","Type":"ContainerStarted","Data":"021dc238d7356b5e1149b28a30970fea1d60b33ac50f13a4ebf58ae7ae0d582c"}
Nov 03 07:05:34.758120 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:05:34.758077    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:05:35.290914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:35.290877    4470 generic.go:356] "Generic (PLEG): container finished" podID="97744a3d-4e77-4c40-a667-2c4e71576378" containerID="021dc238d7356b5e1149b28a30970fea1d60b33ac50f13a4ebf58ae7ae0d582c" exitCode=0
Nov 03 07:05:35.291314 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:35.290942    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm" event={"ID":"97744a3d-4e77-4c40-a667-2c4e71576378","Type":"ContainerDied","Data":"021dc238d7356b5e1149b28a30970fea1d60b33ac50f13a4ebf58ae7ae0d582c"}
Nov 03 07:05:36.295907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:36.295876    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm" event={"ID":"97744a3d-4e77-4c40-a667-2c4e71576378","Type":"ContainerStarted","Data":"10bedd590d38cd95c9334917cf9d4f7dfaa33442ddd74b969ede35c4726520fe"}
Nov 03 07:05:36.296333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:36.295979    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:36.548791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:36.548679    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm" podStartSLOduration=2.681636633 podStartE2EDuration="7.548665834s" totalImagesPullingTime="4.29844893s" totalInitContainerRuntime="568.580271ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:05:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:36.49797112 +0000 UTC m=+388.830340290" watchObservedRunningTime="2026-11-03 07:05:36.548665834 +0000 UTC m=+388.881034822"
Nov 03 07:05:43.540325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.540286    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"]
Nov 03 07:05:43.548287 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.548257    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"
Nov 03 07:05:43.607968 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.607936    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 07:05:43.647395 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.647355    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b75394f0-6c5f-44ab-a691-8d234182f11a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2lbsl\" (UID: \"b75394f0-6c5f-44ab-a691-8d234182f11a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"
Nov 03 07:05:43.647570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.647416    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n46h\" (UniqueName: \"kubernetes.io/projected/b75394f0-6c5f-44ab-a691-8d234182f11a-kube-api-access-2n46h\") pod \"cert-manager-webhook-5dbbbf75cc-2lbsl\" (UID: \"b75394f0-6c5f-44ab-a691-8d234182f11a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"
Nov 03 07:05:43.657520 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.657497    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-d77t4\""
Nov 03 07:05:43.658600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.658579    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"]
Nov 03 07:05:43.708340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.708314    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 07:05:43.748878 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.748852    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n46h\" (UniqueName: \"kubernetes.io/projected/b75394f0-6c5f-44ab-a691-8d234182f11a-kube-api-access-2n46h\") pod \"cert-manager-webhook-5dbbbf75cc-2lbsl\" (UID: \"b75394f0-6c5f-44ab-a691-8d234182f11a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"
Nov 03 07:05:43.749049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.748910    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b75394f0-6c5f-44ab-a691-8d234182f11a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2lbsl\" (UID: \"b75394f0-6c5f-44ab-a691-8d234182f11a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"
Nov 03 07:05:43.814358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.814323    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n46h\" (UniqueName: \"kubernetes.io/projected/b75394f0-6c5f-44ab-a691-8d234182f11a-kube-api-access-2n46h\") pod \"cert-manager-webhook-5dbbbf75cc-2lbsl\" (UID: \"b75394f0-6c5f-44ab-a691-8d234182f11a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"
Nov 03 07:05:43.814497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.814397    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b75394f0-6c5f-44ab-a691-8d234182f11a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2lbsl\" (UID: \"b75394f0-6c5f-44ab-a691-8d234182f11a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"
Nov 03 07:05:43.875717 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:43.875666    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"
Nov 03 07:05:44.258051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:44.257970    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"]
Nov 03 07:05:44.260660 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:05:44.260628    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb75394f0_6c5f_44ab_a691_8d234182f11a.slice/crio-f67bfbb85935c1c642ec999afc310dc4dcc6f492037ef3ed7c71d498ab363470 WatchSource:0}: Error finding container f67bfbb85935c1c642ec999afc310dc4dcc6f492037ef3ed7c71d498ab363470: Status 404 returned error can't find the container with id f67bfbb85935c1c642ec999afc310dc4dcc6f492037ef3ed7c71d498ab363470
Nov 03 07:05:44.325899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:44.325861    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl" event={"ID":"b75394f0-6c5f-44ab-a691-8d234182f11a","Type":"ContainerStarted","Data":"f67bfbb85935c1c642ec999afc310dc4dcc6f492037ef3ed7c71d498ab363470"}
Nov 03 07:05:47.301758 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:47.301731    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-b8ljm"
Nov 03 07:05:48.345280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:48.345196    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl" event={"ID":"b75394f0-6c5f-44ab-a691-8d234182f11a","Type":"ContainerStarted","Data":"898cda65a3e9899d8cfeaa6456cb18864de0fdd7276cf2fa9f30e17d14290354"}
Nov 03 07:05:48.345642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:48.345434    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"
Nov 03 07:05:48.472151 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:48.471920    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl" podStartSLOduration=1.6515394589999999 podStartE2EDuration="5.471900804s" totalImagesPullingTime="3.820361345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:48.471266381 +0000 UTC m=+400.803635402" watchObservedRunningTime="2026-11-03 07:05:48.471900804 +0000 UTC m=+400.804269800"
Nov 03 07:05:51.999762 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:51.999728    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq"]
Nov 03 07:05:52.002885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.002868    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq"
Nov 03 07:05:52.021244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.021210    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e-tmp\") pod \"jobset-operator-674cd59dc5-4v6vq\" (UID: \"bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq"
Nov 03 07:05:52.021367 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.021249    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwgmx\" (UniqueName: \"kubernetes.io/projected/bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e-kube-api-access-zwgmx\") pod \"jobset-operator-674cd59dc5-4v6vq\" (UID: \"bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq"
Nov 03 07:05:52.108219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.108188    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:05:52.108219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.108202    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-mbb5n\""
Nov 03 07:05:52.108429 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.108246    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 07:05:52.109366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.109348    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq"]
Nov 03 07:05:52.121898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.121878    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e-tmp\") pod \"jobset-operator-674cd59dc5-4v6vq\" (UID: \"bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq"
Nov 03 07:05:52.121945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.121908    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwgmx\" (UniqueName: \"kubernetes.io/projected/bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e-kube-api-access-zwgmx\") pod \"jobset-operator-674cd59dc5-4v6vq\" (UID: \"bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq"
Nov 03 07:05:52.127405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.122732    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e-tmp\") pod \"jobset-operator-674cd59dc5-4v6vq\" (UID: \"bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq"
Nov 03 07:05:52.192229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.192192    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwgmx\" (UniqueName: \"kubernetes.io/projected/bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e-kube-api-access-zwgmx\") pod \"jobset-operator-674cd59dc5-4v6vq\" (UID: \"bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq"
Nov 03 07:05:52.314353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.314323    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq"
Nov 03 07:05:52.697888 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:52.697674    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq"]
Nov 03 07:05:52.700479 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:05:52.700451    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfe7075c_54bd_4d95_8f22_ddb09d1e6b7e.slice/crio-127bde629a1888a340939cae348a3ebc56104f2f67c1a1e087860320b6ab12a4 WatchSource:0}: Error finding container 127bde629a1888a340939cae348a3ebc56104f2f67c1a1e087860320b6ab12a4: Status 404 returned error can't find the container with id 127bde629a1888a340939cae348a3ebc56104f2f67c1a1e087860320b6ab12a4
Nov 03 07:05:53.367114 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:53.367079    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq" event={"ID":"bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e","Type":"ContainerStarted","Data":"127bde629a1888a340939cae348a3ebc56104f2f67c1a1e087860320b6ab12a4"}
Nov 03 07:05:54.352102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:54.352067    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2lbsl"
Nov 03 07:05:55.367030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:55.367000    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-k8sfp"]
Nov 03 07:05:55.375633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:55.375610    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-k8sfp"
Nov 03 07:05:55.449431 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:55.449395    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwljc\" (UniqueName: \"kubernetes.io/projected/57319368-235d-4dbb-89a0-2186aca638be-kube-api-access-jwljc\") pod \"cert-manager-597b95cb84-k8sfp\" (UID: \"57319368-235d-4dbb-89a0-2186aca638be\") " pod="cert-manager/cert-manager-597b95cb84-k8sfp"
Nov 03 07:05:55.449613 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:55.449534    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57319368-235d-4dbb-89a0-2186aca638be-bound-sa-token\") pod \"cert-manager-597b95cb84-k8sfp\" (UID: \"57319368-235d-4dbb-89a0-2186aca638be\") " pod="cert-manager/cert-manager-597b95cb84-k8sfp"
Nov 03 07:05:55.483299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:55.483266    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-kpnv7\""
Nov 03 07:05:55.484601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:55.484576    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-k8sfp"]
Nov 03 07:05:55.550636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:55.550595    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57319368-235d-4dbb-89a0-2186aca638be-bound-sa-token\") pod \"cert-manager-597b95cb84-k8sfp\" (UID: \"57319368-235d-4dbb-89a0-2186aca638be\") " pod="cert-manager/cert-manager-597b95cb84-k8sfp"
Nov 03 07:05:55.550801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:55.550698    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwljc\" (UniqueName: \"kubernetes.io/projected/57319368-235d-4dbb-89a0-2186aca638be-kube-api-access-jwljc\") pod \"cert-manager-597b95cb84-k8sfp\" (UID: \"57319368-235d-4dbb-89a0-2186aca638be\") " pod="cert-manager/cert-manager-597b95cb84-k8sfp"
Nov 03 07:05:55.627059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:55.626992    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwljc\" (UniqueName: \"kubernetes.io/projected/57319368-235d-4dbb-89a0-2186aca638be-kube-api-access-jwljc\") pod \"cert-manager-597b95cb84-k8sfp\" (UID: \"57319368-235d-4dbb-89a0-2186aca638be\") " pod="cert-manager/cert-manager-597b95cb84-k8sfp"
Nov 03 07:05:55.627231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:55.627093    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57319368-235d-4dbb-89a0-2186aca638be-bound-sa-token\") pod \"cert-manager-597b95cb84-k8sfp\" (UID: \"57319368-235d-4dbb-89a0-2186aca638be\") " pod="cert-manager/cert-manager-597b95cb84-k8sfp"
Nov 03 07:05:55.688321 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:55.688284    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-k8sfp"
Nov 03 07:05:56.080925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:56.080898    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-k8sfp"]
Nov 03 07:05:56.083915 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:05:56.083878    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57319368_235d_4dbb_89a0_2186aca638be.slice/crio-a12130b4abbc3c53c730f7e6a9fe81911f972c20a4b953fef2108cd7aef8bac2 WatchSource:0}: Error finding container a12130b4abbc3c53c730f7e6a9fe81911f972c20a4b953fef2108cd7aef8bac2: Status 404 returned error can't find the container with id a12130b4abbc3c53c730f7e6a9fe81911f972c20a4b953fef2108cd7aef8bac2
Nov 03 07:05:56.384883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:56.384804    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-k8sfp" event={"ID":"57319368-235d-4dbb-89a0-2186aca638be","Type":"ContainerStarted","Data":"69699a349d3b1e87aeeb123e508d78b91a20b5a5212a8a00e0fafafe288e9b77"}
Nov 03 07:05:56.384883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:56.384848    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-k8sfp" event={"ID":"57319368-235d-4dbb-89a0-2186aca638be","Type":"ContainerStarted","Data":"a12130b4abbc3c53c730f7e6a9fe81911f972c20a4b953fef2108cd7aef8bac2"}
Nov 03 07:05:56.516801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:56.516742    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-k8sfp" podStartSLOduration=1.516729913 podStartE2EDuration="1.516729913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:56.516591341 +0000 UTC m=+408.848960354" watchObservedRunningTime="2026-11-03 07:05:56.516729913 +0000 UTC m=+408.849098908"
Nov 03 07:05:57.391106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:57.391070    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq" event={"ID":"bfe7075c-54bd-4d95-8f22-ddb09d1e6b7e","Type":"ContainerStarted","Data":"56e9e7ed93e1d299f17d1a944643d7a27cc49b3aad6c3615be27e069c948552f"}
Nov 03 07:05:57.513221 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:05:57.513169    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4v6vq" podStartSLOduration=2.827930352 podStartE2EDuration="6.513154564s" totalImagesPullingTime="3.685224212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:57.512912731 +0000 UTC m=+409.845281725" watchObservedRunningTime="2026-11-03 07:05:57.513154564 +0000 UTC m=+409.845523559"
Nov 03 07:06:04.759479 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:06:04.759439    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:06:09.371497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.371457    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"]
Nov 03 07:06:09.376219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.376197    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"
Nov 03 07:06:09.434245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.434211    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 07:06:09.434491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.434273    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 07:06:09.434760 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.434685    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"]
Nov 03 07:06:09.481480 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.481441    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7hwl\" (UniqueName: \"kubernetes.io/projected/217855d1-2dcc-476d-8b22-0fc92ff7f242-kube-api-access-m7hwl\") pod \"kube-auth-proxy-86b9d545b-jsxrj\" (UID: \"217855d1-2dcc-476d-8b22-0fc92ff7f242\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"
Nov 03 07:06:09.481659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.481596    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217855d1-2dcc-476d-8b22-0fc92ff7f242-tls-certs\") pod \"kube-auth-proxy-86b9d545b-jsxrj\" (UID: \"217855d1-2dcc-476d-8b22-0fc92ff7f242\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"
Nov 03 07:06:09.481659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.481619    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/217855d1-2dcc-476d-8b22-0fc92ff7f242-tmp\") pod \"kube-auth-proxy-86b9d545b-jsxrj\" (UID: \"217855d1-2dcc-476d-8b22-0fc92ff7f242\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"
Nov 03 07:06:09.483299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.483275    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 07:06:09.483466 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.483378    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 07:06:09.483553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.483499    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-skt9p\""
Nov 03 07:06:09.582924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.582886    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217855d1-2dcc-476d-8b22-0fc92ff7f242-tls-certs\") pod \"kube-auth-proxy-86b9d545b-jsxrj\" (UID: \"217855d1-2dcc-476d-8b22-0fc92ff7f242\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"
Nov 03 07:06:09.582924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.582925    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/217855d1-2dcc-476d-8b22-0fc92ff7f242-tmp\") pod \"kube-auth-proxy-86b9d545b-jsxrj\" (UID: \"217855d1-2dcc-476d-8b22-0fc92ff7f242\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"
Nov 03 07:06:09.583205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.582988    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7hwl\" (UniqueName: \"kubernetes.io/projected/217855d1-2dcc-476d-8b22-0fc92ff7f242-kube-api-access-m7hwl\") pod \"kube-auth-proxy-86b9d545b-jsxrj\" (UID: \"217855d1-2dcc-476d-8b22-0fc92ff7f242\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"
Nov 03 07:06:09.585897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.585873    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/217855d1-2dcc-476d-8b22-0fc92ff7f242-tmp\") pod \"kube-auth-proxy-86b9d545b-jsxrj\" (UID: \"217855d1-2dcc-476d-8b22-0fc92ff7f242\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"
Nov 03 07:06:09.586128 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.586106    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217855d1-2dcc-476d-8b22-0fc92ff7f242-tls-certs\") pod \"kube-auth-proxy-86b9d545b-jsxrj\" (UID: \"217855d1-2dcc-476d-8b22-0fc92ff7f242\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"
Nov 03 07:06:09.647074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.647003    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7hwl\" (UniqueName: \"kubernetes.io/projected/217855d1-2dcc-476d-8b22-0fc92ff7f242-kube-api-access-m7hwl\") pod \"kube-auth-proxy-86b9d545b-jsxrj\" (UID: \"217855d1-2dcc-476d-8b22-0fc92ff7f242\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"
Nov 03 07:06:09.687766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:09.687733    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"
Nov 03 07:06:10.067253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:10.067228    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj"]
Nov 03 07:06:10.069065 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:06:10.069030    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217855d1_2dcc_476d_8b22_0fc92ff7f242.slice/crio-338db77e8184ab28658acdaad5062c0398558ddc575119f9e9d3b848a5f24694 WatchSource:0}: Error finding container 338db77e8184ab28658acdaad5062c0398558ddc575119f9e9d3b848a5f24694: Status 404 returned error can't find the container with id 338db77e8184ab28658acdaad5062c0398558ddc575119f9e9d3b848a5f24694
Nov 03 07:06:10.443270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:10.443191    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj" event={"ID":"217855d1-2dcc-476d-8b22-0fc92ff7f242","Type":"ContainerStarted","Data":"338db77e8184ab28658acdaad5062c0398558ddc575119f9e9d3b848a5f24694"}
Nov 03 07:06:13.458068 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:13.458034    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj" event={"ID":"217855d1-2dcc-476d-8b22-0fc92ff7f242","Type":"ContainerStarted","Data":"b11979f256f4b7f9bbc3ecdb7887b5c10651dd75581425dde038d38d15b0aaad"}
Nov 03 07:06:13.580449 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:13.580389    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj" podStartSLOduration=1.662999734 podStartE2EDuration="4.580374636s" totalImagesPullingTime="2.917374902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:06:13.579576094 +0000 UTC m=+425.911945089" watchObservedRunningTime="2026-11-03 07:06:13.580374636 +0000 UTC m=+425.912743631"
Nov 03 07:06:17.586974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.586942    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"]
Nov 03 07:06:17.633714 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.633690    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.650363 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.650312    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"]
Nov 03 07:06:17.699410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.699376    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 07:06:17.699410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.699379    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-pth4w\""
Nov 03 07:06:17.738709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.738677    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 07:06:17.738846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.738724    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 07:06:17.757888 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.757863    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c81aa125-5d40-4493-81c4-e2a99db9278c-metrics-certs\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.757993 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.757902    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkgz\" (UniqueName: \"kubernetes.io/projected/c81aa125-5d40-4493-81c4-e2a99db9278c-kube-api-access-9dkgz\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.758036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.758016    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c81aa125-5d40-4493-81c4-e2a99db9278c-manager-config\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.758070 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.758041    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c81aa125-5d40-4493-81c4-e2a99db9278c-cert\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.858904 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.858831    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c81aa125-5d40-4493-81c4-e2a99db9278c-manager-config\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.858904 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.858864    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c81aa125-5d40-4493-81c4-e2a99db9278c-cert\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.858904 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.858900    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c81aa125-5d40-4493-81c4-e2a99db9278c-metrics-certs\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.859380 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.858968    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkgz\" (UniqueName: \"kubernetes.io/projected/c81aa125-5d40-4493-81c4-e2a99db9278c-kube-api-access-9dkgz\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.859641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.859621    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c81aa125-5d40-4493-81c4-e2a99db9278c-manager-config\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.861378 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.861360    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c81aa125-5d40-4493-81c4-e2a99db9278c-cert\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.861428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.861393    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c81aa125-5d40-4493-81c4-e2a99db9278c-metrics-certs\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.927045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.927015    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkgz\" (UniqueName: \"kubernetes.io/projected/c81aa125-5d40-4493-81c4-e2a99db9278c-kube-api-access-9dkgz\") pod \"jobset-controller-manager-8594984bd6-9vcx6\" (UID: \"c81aa125-5d40-4493-81c4-e2a99db9278c\") " pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:17.963368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:17.963334    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:18.365183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:18.365159    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"]
Nov 03 07:06:18.367210 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:06:18.367171    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81aa125_5d40_4493_81c4_e2a99db9278c.slice/crio-2a61b64d6111d9efcc62cd8b5978f46e64aa3eb74b656b30463782a7c457c9d6 WatchSource:0}: Error finding container 2a61b64d6111d9efcc62cd8b5978f46e64aa3eb74b656b30463782a7c457c9d6: Status 404 returned error can't find the container with id 2a61b64d6111d9efcc62cd8b5978f46e64aa3eb74b656b30463782a7c457c9d6
Nov 03 07:06:18.479955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:18.479918    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6" event={"ID":"c81aa125-5d40-4493-81c4-e2a99db9278c","Type":"ContainerStarted","Data":"2a61b64d6111d9efcc62cd8b5978f46e64aa3eb74b656b30463782a7c457c9d6"}
Nov 03 07:06:23.505277 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:23.505241    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6" event={"ID":"c81aa125-5d40-4493-81c4-e2a99db9278c","Type":"ContainerStarted","Data":"449e6748d5b7fd43f84b65e89acca0d3cdb41b48dabc88c604f362b31d9cf5a9"}
Nov 03 07:06:23.505695 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:23.505364    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:23.626506 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:23.626451    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6" podStartSLOduration=2.358776707 podStartE2EDuration="6.62643842s" totalImagesPullingTime="4.267661713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:06:23.625485996 +0000 UTC m=+435.957854988" watchObservedRunningTime="2026-11-03 07:06:23.62643842 +0000 UTC m=+435.958807413"
Nov 03 07:06:34.513687 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:06:34.513651    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-8594984bd6-9vcx6"
Nov 03 07:06:34.757358 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:06:34.757322    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:07:04.758286 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:07:04.758247    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:07:30.917552 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:30.917479    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pj94l"]
Nov 03 07:07:30.920679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:30.920661    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l"
Nov 03 07:07:30.980947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:30.980915    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pj94l"]
Nov 03 07:07:30.999880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:30.999848    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f544e17f-b5e1-42d4-9e4a-fd50df088f26-cert\") pod \"kserve-controller-manager-776cc66d47-pj94l\" (UID: \"f544e17f-b5e1-42d4-9e4a-fd50df088f26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l"
Nov 03 07:07:31.000028 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:30.999892    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pdzj\" (UniqueName: \"kubernetes.io/projected/f544e17f-b5e1-42d4-9e4a-fd50df088f26-kube-api-access-5pdzj\") pod \"kserve-controller-manager-776cc66d47-pj94l\" (UID: \"f544e17f-b5e1-42d4-9e4a-fd50df088f26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l"
Nov 03 07:07:31.029696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:31.029661    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vbnvb\""
Nov 03 07:07:31.029871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:31.029740    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 07:07:31.029871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:31.029740    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 07:07:31.029871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:31.029763    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 07:07:31.100749 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:31.100715    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f544e17f-b5e1-42d4-9e4a-fd50df088f26-cert\") pod \"kserve-controller-manager-776cc66d47-pj94l\" (UID: \"f544e17f-b5e1-42d4-9e4a-fd50df088f26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l"
Nov 03 07:07:31.100901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:31.100759    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdzj\" (UniqueName: \"kubernetes.io/projected/f544e17f-b5e1-42d4-9e4a-fd50df088f26-kube-api-access-5pdzj\") pod \"kserve-controller-manager-776cc66d47-pj94l\" (UID: \"f544e17f-b5e1-42d4-9e4a-fd50df088f26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l"
Nov 03 07:07:31.103321 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:31.103303    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f544e17f-b5e1-42d4-9e4a-fd50df088f26-cert\") pod \"kserve-controller-manager-776cc66d47-pj94l\" (UID: \"f544e17f-b5e1-42d4-9e4a-fd50df088f26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l"
Nov 03 07:07:31.165280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:31.165245    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pdzj\" (UniqueName: \"kubernetes.io/projected/f544e17f-b5e1-42d4-9e4a-fd50df088f26-kube-api-access-5pdzj\") pod \"kserve-controller-manager-776cc66d47-pj94l\" (UID: \"f544e17f-b5e1-42d4-9e4a-fd50df088f26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l"
Nov 03 07:07:31.232252 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:31.232177    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l"
Nov 03 07:07:31.604779 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:31.604723    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pj94l"]
Nov 03 07:07:31.607057 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:07:31.607029    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf544e17f_b5e1_42d4_9e4a_fd50df088f26.slice/crio-cf88c4f6c1b979ad125c821a4fee09fd829a442e1e49cb6eab2b6db3d7a6d2ab WatchSource:0}: Error finding container cf88c4f6c1b979ad125c821a4fee09fd829a442e1e49cb6eab2b6db3d7a6d2ab: Status 404 returned error can't find the container with id cf88c4f6c1b979ad125c821a4fee09fd829a442e1e49cb6eab2b6db3d7a6d2ab
Nov 03 07:07:31.793546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:31.793515    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l" event={"ID":"f544e17f-b5e1-42d4-9e4a-fd50df088f26","Type":"ContainerStarted","Data":"cf88c4f6c1b979ad125c821a4fee09fd829a442e1e49cb6eab2b6db3d7a6d2ab"}
Nov 03 07:07:34.758883 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:07:34.758846    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:07:35.814107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:35.814074    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l" event={"ID":"f544e17f-b5e1-42d4-9e4a-fd50df088f26","Type":"ContainerStarted","Data":"db149a48a988270ee485ac91510cf37c5f1c8379bf38174f9829816f5bbc2c0e"}
Nov 03 07:07:35.814507 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:35.814167    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l"
Nov 03 07:07:35.936059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:07:35.936007    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l" podStartSLOduration=2.778646353 podStartE2EDuration="5.935992024s" totalImagesPullingTime="3.157345671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:07:35.935025961 +0000 UTC m=+508.267394957" watchObservedRunningTime="2026-11-03 07:07:35.935992024 +0000 UTC m=+508.268361018"
Nov 03 07:08:04.758482 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:08:04.758440    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:08:06.823731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:08:06.823705    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pj94l"
Nov 03 07:08:34.759022 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:08:34.758919    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:09:04.757877 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:09:04.757843    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:09:34.758395 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:09:34.758359    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:09:35.667355 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.667320    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"]
Nov 03 07:09:35.670690 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.670675    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.776012 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.775976    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 07:09:35.776012 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.775988    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vdz8z\""
Nov 03 07:09:35.776012 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.776011    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 07:09:35.776505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.776043    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 07:09:35.777569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.777538    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"]
Nov 03 07:09:35.797651 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.797622    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.797824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.797672    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.797824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.797743    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.797910 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.797849    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bqjf\" (UniqueName: \"kubernetes.io/projected/69402d62-2b20-4b8b-b0e8-ba113180fcf7-kube-api-access-6bqjf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.797910 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.797889    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69402d62-2b20-4b8b-b0e8-ba113180fcf7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.797981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.797909    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.898958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.898916    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqjf\" (UniqueName: \"kubernetes.io/projected/69402d62-2b20-4b8b-b0e8-ba113180fcf7-kube-api-access-6bqjf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.899178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.898977    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69402d62-2b20-4b8b-b0e8-ba113180fcf7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.899178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.899005    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.899178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.899038    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.899178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.899097    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.899178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.899162    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.899473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.899447    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.899654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.899506    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.899741 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.899577    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.901493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.901474    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69402d62-2b20-4b8b-b0e8-ba113180fcf7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.901768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.901751    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69402d62-2b20-4b8b-b0e8-ba113180fcf7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.964834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.964766    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bqjf\" (UniqueName: \"kubernetes.io/projected/69402d62-2b20-4b8b-b0e8-ba113180fcf7-kube-api-access-6bqjf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm\" (UID: \"69402d62-2b20-4b8b-b0e8-ba113180fcf7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:35.983241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:35.983208    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:36.359382 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:36.359358    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"]
Nov 03 07:09:36.361109 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:09:36.361086    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69402d62_2b20_4b8b_b0e8_ba113180fcf7.slice/crio-2e48fa35f9069ff17f3922b5ae4bfca89266af77256848a119ca867a3c5d66b0 WatchSource:0}: Error finding container 2e48fa35f9069ff17f3922b5ae4bfca89266af77256848a119ca867a3c5d66b0: Status 404 returned error can't find the container with id 2e48fa35f9069ff17f3922b5ae4bfca89266af77256848a119ca867a3c5d66b0
Nov 03 07:09:37.302552 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:37.302509    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm" event={"ID":"69402d62-2b20-4b8b-b0e8-ba113180fcf7","Type":"ContainerStarted","Data":"2e48fa35f9069ff17f3922b5ae4bfca89266af77256848a119ca867a3c5d66b0"}
Nov 03 07:09:39.312970 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:39.312936    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm" event={"ID":"69402d62-2b20-4b8b-b0e8-ba113180fcf7","Type":"ContainerStarted","Data":"f4512f7376ee53943869a2f2f027c60952d7fae3b7bce9a8f3600ee880ee05f2"}
Nov 03 07:09:39.438065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:39.438011    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm" podStartSLOduration=2.375749597 podStartE2EDuration="4.437997645s" totalImagesPullingTime="2.062248048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:09:39.43693575 +0000 UTC m=+631.769304769" watchObservedRunningTime="2026-11-03 07:09:39.437997645 +0000 UTC m=+631.770366640"
Nov 03 07:09:39.984228 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:39.984193    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:45.984343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:45.984297    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:45.996732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:45.996711    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:46.353700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:46.353668    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm"
Nov 03 07:09:47.525390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.525358    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"]
Nov 03 07:09:47.531612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.531594    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.589653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.589621    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"]
Nov 03 07:09:47.637745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.637715    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 07:09:47.707768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.707738    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.707924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.707783    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.707924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.707862    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.707924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.707891    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtbxv\" (UniqueName: \"kubernetes.io/projected/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-kube-api-access-vtbxv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.708034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.707931    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.708034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.707974    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.808916 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.808832    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.808916 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.808895    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.809179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.808942    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.809179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.808963    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtbxv\" (UniqueName: \"kubernetes.io/projected/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-kube-api-access-vtbxv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.809179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.809015    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.809179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.809038    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.809408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.809381    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.809537 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.809387    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.809537 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.809476    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.811434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.811415    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.811487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.811420    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:47.873569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:47.873534    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtbxv\" (UniqueName: \"kubernetes.io/projected/ff22aa3b-6c14-43ef-8658-1d6a6d13c53c-kube-api-access-vtbxv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc\" (UID: \"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:48.143078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:48.143042    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:48.517929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:48.517905    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"]
Nov 03 07:09:48.519895 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:09:48.519869    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff22aa3b_6c14_43ef_8658_1d6a6d13c53c.slice/crio-105aeab5e7585966660648ba7b4dfd2c37ba27380a9c909a9f664220b52e2930 WatchSource:0}: Error finding container 105aeab5e7585966660648ba7b4dfd2c37ba27380a9c909a9f664220b52e2930: Status 404 returned error can't find the container with id 105aeab5e7585966660648ba7b4dfd2c37ba27380a9c909a9f664220b52e2930
Nov 03 07:09:49.358209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:49.358181    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" event={"ID":"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c","Type":"ContainerStarted","Data":"3ecc4aaa08184f3de02f6248c673c2e4e152f8a08d2d29e552d53dac23e94c4a"}
Nov 03 07:09:49.358585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:49.358216    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" event={"ID":"ff22aa3b-6c14-43ef-8658-1d6a6d13c53c","Type":"ContainerStarted","Data":"105aeab5e7585966660648ba7b4dfd2c37ba27380a9c909a9f664220b52e2930"}
Nov 03 07:09:49.481298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:49.481197    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podStartSLOduration=1.887481831 podStartE2EDuration="2.48118429s" totalImagesPullingTime="593.702459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:09:49.480226829 +0000 UTC m=+641.812595824" watchObservedRunningTime="2026-11-03 07:09:49.48118429 +0000 UTC m=+641.813553284"
Nov 03 07:09:50.143397 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:50.143359    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:58.143919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.143878    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:58.156294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.156272    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:58.411734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.411658    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:09:58.680093 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.680015    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"]
Nov 03 07:09:58.683897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.683877    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.704320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.704299    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.704424 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.704339    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5975\" (UniqueName: \"kubernetes.io/projected/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-kube-api-access-v5975\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.704424 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.704370    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.704424 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.704406    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.704528 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.704511    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.704566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.704553    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.741098 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.741062    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 07:09:58.747066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.747036    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"]
Nov 03 07:09:58.805024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.804988    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.805225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.805040    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.805225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.805075    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.805225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.805105    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5975\" (UniqueName: \"kubernetes.io/projected/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-kube-api-access-v5975\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.805225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.805153    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.805225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.805168    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.806044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.805999    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.806044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.806033    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.806259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.806238    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.807766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.807746    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.808207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.808190    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.870674 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.870637    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5975\" (UniqueName: \"kubernetes.io/projected/bdedcd4a-7850-416b-906c-8bffd4b5a2ef-kube-api-access-v5975\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ffxdf\" (UID: \"bdedcd4a-7850-416b-906c-8bffd4b5a2ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:58.996477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:58.996399    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:09:59.369855 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:59.369832    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"]
Nov 03 07:09:59.371978 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:09:59.371951    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdedcd4a_7850_416b_906c_8bffd4b5a2ef.slice/crio-9930ba1c7c9ed2ead635f68b2b8ac55ae059f49c719ef6005c01d32648382ed3 WatchSource:0}: Error finding container 9930ba1c7c9ed2ead635f68b2b8ac55ae059f49c719ef6005c01d32648382ed3: Status 404 returned error can't find the container with id 9930ba1c7c9ed2ead635f68b2b8ac55ae059f49c719ef6005c01d32648382ed3
Nov 03 07:09:59.402080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:09:59.402053    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf" event={"ID":"bdedcd4a-7850-416b-906c-8bffd4b5a2ef","Type":"ContainerStarted","Data":"9930ba1c7c9ed2ead635f68b2b8ac55ae059f49c719ef6005c01d32648382ed3"}
Nov 03 07:10:00.407996 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:00.407963    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf" event={"ID":"bdedcd4a-7850-416b-906c-8bffd4b5a2ef","Type":"ContainerStarted","Data":"9de4ad9453570409a6ddbc68c14c76c75a90b518e5684350bfc5becfe130102e"}
Nov 03 07:10:00.530648 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:00.530550    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf" podStartSLOduration=1.716489895 podStartE2EDuration="2.530535742s" totalImagesPullingTime="814.045847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:10:00.530170987 +0000 UTC m=+652.862539975" watchObservedRunningTime="2026-11-03 07:10:00.530535742 +0000 UTC m=+652.862904737"
Nov 03 07:10:00.997228 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:00.997180    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:10:02.012959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.012928    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"]
Nov 03 07:10:02.016636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.016615    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.034715 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.034686    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.034876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.034731    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg5mc\" (UniqueName: \"kubernetes.io/projected/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-kube-api-access-pg5mc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.034876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.034763    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.034876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.034858    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.035021 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.034887    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.035021 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.034901    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.081181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.081147    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"]
Nov 03 07:10:02.126626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.126592    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 07:10:02.135460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.135426    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.135627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.135467    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.135627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.135485    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.135627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.135533    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.135627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.135559    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg5mc\" (UniqueName: \"kubernetes.io/projected/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-kube-api-access-pg5mc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.135627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.135576    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.135907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.135882    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.135981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.135902    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.135981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.135936    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.138021 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.137994    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.138192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.138175    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.199988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.199953    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg5mc\" (UniqueName: \"kubernetes.io/projected/506a0cd0-0eb3-49d8-b5bf-8c841e823d10-kube-api-access-pg5mc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mkrz\" (UID: \"506a0cd0-0eb3-49d8-b5bf-8c841e823d10\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.329649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.329614    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:02.709018 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:02.708994    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"]
Nov 03 07:10:02.711300 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:10:02.711275    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506a0cd0_0eb3_49d8_b5bf_8c841e823d10.slice/crio-84369a2ff48d836f66c05ae9450e8e004c4d8e81e55b127d59269300aea5a27c WatchSource:0}: Error finding container 84369a2ff48d836f66c05ae9450e8e004c4d8e81e55b127d59269300aea5a27c: Status 404 returned error can't find the container with id 84369a2ff48d836f66c05ae9450e8e004c4d8e81e55b127d59269300aea5a27c
Nov 03 07:10:03.424282 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:03.424240    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz" event={"ID":"506a0cd0-0eb3-49d8-b5bf-8c841e823d10","Type":"ContainerStarted","Data":"84369a2ff48d836f66c05ae9450e8e004c4d8e81e55b127d59269300aea5a27c"}
Nov 03 07:10:04.430434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:04.430394    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz" event={"ID":"506a0cd0-0eb3-49d8-b5bf-8c841e823d10","Type":"ContainerStarted","Data":"ba7b6401d4d899c95b55f768c50c7acbd5dee8bbf6f09256514d1f94c8efadf5"}
Nov 03 07:10:04.553888 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:04.553824    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz" podStartSLOduration=2.871724612 podStartE2EDuration="3.553809946s" totalImagesPullingTime="682.085334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:10:04.552534408 +0000 UTC m=+656.884903402" watchObservedRunningTime="2026-11-03 07:10:04.553809946 +0000 UTC m=+656.886178941"
Nov 03 07:10:04.758621 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:10:04.758539    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:10:05.330222 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:05.330190    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:08.997177 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:08.997112    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:10:09.009899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:09.009866    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:10:09.461630 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:09.461605    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ffxdf"
Nov 03 07:10:12.028977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.028935    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"]
Nov 03 07:10:12.032572 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.032547    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.087039 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.087008    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89ggc\""
Nov 03 07:10:12.132675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.132645    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.132828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.132681    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.132828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.132706    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.132828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.132776    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/155e6cf3-4920-4d89-b783-7125c6974485-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.132926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.132830    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/155e6cf3-4920-4d89-b783-7125c6974485-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.132926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.132878    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.132926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.132917    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.133020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.132941    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2p49\" (UniqueName: \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-kube-api-access-x2p49\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.133053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.133025    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.137977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.137952    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 07:10:12.140496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.140477    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"]
Nov 03 07:10:12.234038 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234007    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/155e6cf3-4920-4d89-b783-7125c6974485-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.234246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234074    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/155e6cf3-4920-4d89-b783-7125c6974485-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.234246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234098    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.234246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234157    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.234246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234192    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2p49\" (UniqueName: \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-kube-api-access-x2p49\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.234246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234212    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.234510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234249    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.234510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234274    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.234510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234299    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.234696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234629    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.234696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234657    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.234970 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.234948    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.235092 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.235048    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.235092 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.235077    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/155e6cf3-4920-4d89-b783-7125c6974485-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.236710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.236689    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/155e6cf3-4920-4d89-b783-7125c6974485-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.236798 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.236708    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.298249 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.298180    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.298249 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.298234    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2p49\" (UniqueName: \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-kube-api-access-x2p49\") pod \"maas-default-gateway-openshift-default-58d898bd55-p5gnc\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.329722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.329697    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:12.342572 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.342545    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:12.344690 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.344671    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:12.477323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.477295    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mkrz"
Nov 03 07:10:12.771865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:12.771840    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"]
Nov 03 07:10:12.773628 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:10:12.773597    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155e6cf3_4920_4d89_b783_7125c6974485.slice/crio-ebc4a6e8081a77d2800f94358240f96f5fb60ecbc7fadaa651938e2fa7471e80 WatchSource:0}: Error finding container ebc4a6e8081a77d2800f94358240f96f5fb60ecbc7fadaa651938e2fa7471e80: Status 404 returned error can't find the container with id ebc4a6e8081a77d2800f94358240f96f5fb60ecbc7fadaa651938e2fa7471e80
Nov 03 07:10:13.471614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:13.471578    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" event={"ID":"155e6cf3-4920-4d89-b783-7125c6974485","Type":"ContainerStarted","Data":"ebc4a6e8081a77d2800f94358240f96f5fb60ecbc7fadaa651938e2fa7471e80"}
Nov 03 07:10:16.292708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:16.292676    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:10:16.292970 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:16.292745    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:10:16.292970 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:16.292778    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:10:16.490944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:16.490908    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" event={"ID":"155e6cf3-4920-4d89-b783-7125c6974485","Type":"ContainerStarted","Data":"b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21"}
Nov 03 07:10:16.627360 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:16.627305    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" podStartSLOduration=2.111197888 podStartE2EDuration="5.627293151s" totalImagesPullingTime="3.516095263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:10:16.626358421 +0000 UTC m=+668.958727427" watchObservedRunningTime="2026-11-03 07:10:16.627293151 +0000 UTC m=+668.959662184"
Nov 03 07:10:17.345213 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:17.345172    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:17.371184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:17.365713    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 07:10:17.371184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:17.365819    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" podUID="155e6cf3-4920-4d89-b783-7125c6974485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 07:10:18.072049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.072006    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-zmd8p"]
Nov 03 07:10:18.076847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.076822    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:18.139952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.139917    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-zmd8p"]
Nov 03 07:10:18.188377 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.188338    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-v9t2w\""
Nov 03 07:10:18.188609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.188460    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 07:10:18.188609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.188526    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 07:10:18.191183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.191159    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmnp4\" (UniqueName: \"kubernetes.io/projected/905054cb-da87-4246-ae0a-5f02a8793111-kube-api-access-hmnp4\") pod \"ai-gateway-controller-86c7698f44-zmd8p\" (UID: \"905054cb-da87-4246-ae0a-5f02a8793111\") " pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:18.191269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.191196    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/905054cb-da87-4246-ae0a-5f02a8793111-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-zmd8p\" (UID: \"905054cb-da87-4246-ae0a-5f02a8793111\") " pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:18.292932 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.292894    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmnp4\" (UniqueName: \"kubernetes.io/projected/905054cb-da87-4246-ae0a-5f02a8793111-kube-api-access-hmnp4\") pod \"ai-gateway-controller-86c7698f44-zmd8p\" (UID: \"905054cb-da87-4246-ae0a-5f02a8793111\") " pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:18.293105 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.292951    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/905054cb-da87-4246-ae0a-5f02a8793111-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-zmd8p\" (UID: \"905054cb-da87-4246-ae0a-5f02a8793111\") " pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:18.295713 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.295686    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/905054cb-da87-4246-ae0a-5f02a8793111-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-zmd8p\" (UID: \"905054cb-da87-4246-ae0a-5f02a8793111\") " pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:18.345182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.345063    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 07:10:18.345182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.345148    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" podUID="155e6cf3-4920-4d89-b783-7125c6974485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 07:10:18.377787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.377756    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmnp4\" (UniqueName: \"kubernetes.io/projected/905054cb-da87-4246-ae0a-5f02a8793111-kube-api-access-hmnp4\") pod \"ai-gateway-controller-86c7698f44-zmd8p\" (UID: \"905054cb-da87-4246-ae0a-5f02a8793111\") " pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:18.389635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.389598    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:18.795730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:18.795708    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-zmd8p"]
Nov 03 07:10:18.797342 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:10:18.797311    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905054cb_da87_4246_ae0a_5f02a8793111.slice/crio-db0292c4055d7a91c5b73331a6799264107c8a87d38bb2b50e0790ac56b6f2a0 WatchSource:0}: Error finding container db0292c4055d7a91c5b73331a6799264107c8a87d38bb2b50e0790ac56b6f2a0: Status 404 returned error can't find the container with id db0292c4055d7a91c5b73331a6799264107c8a87d38bb2b50e0790ac56b6f2a0
Nov 03 07:10:19.344963 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:19.344927    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 07:10:19.345159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:19.344973    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" podUID="155e6cf3-4920-4d89-b783-7125c6974485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 07:10:19.505484 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:19.505446    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p" event={"ID":"905054cb-da87-4246-ae0a-5f02a8793111","Type":"ContainerStarted","Data":"db0292c4055d7a91c5b73331a6799264107c8a87d38bb2b50e0790ac56b6f2a0"}
Nov 03 07:10:20.345783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:20.345749    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 07:10:20.345960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:20.345831    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" podUID="155e6cf3-4920-4d89-b783-7125c6974485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 07:10:21.345494 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:21.345330    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 07:10:21.345494 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:21.345388    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" podUID="155e6cf3-4920-4d89-b783-7125c6974485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 07:10:22.345924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:22.345884    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 07:10:22.346336 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:22.345959    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" podUID="155e6cf3-4920-4d89-b783-7125c6974485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 07:10:23.345473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:23.345439    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 07:10:23.345664 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:23.345498    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" podUID="155e6cf3-4920-4d89-b783-7125c6974485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 07:10:24.592671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:24.592614    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:24.593234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:24.593203    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:24.594256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:24.594234    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:10:25.542398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:25.542366    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p" event={"ID":"905054cb-da87-4246-ae0a-5f02a8793111","Type":"ContainerStarted","Data":"37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a"}
Nov 03 07:10:25.542603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:25.542578    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:25.664011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:25.663956    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p" podStartSLOduration=1.512584232 podStartE2EDuration="7.663940811s" totalImagesPullingTime="6.151356579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:10:25.66360339 +0000 UTC m=+677.995972382" watchObservedRunningTime="2026-11-03 07:10:25.663940811 +0000 UTC m=+677.996309805"
Nov 03 07:10:28.221411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.221374    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"]
Nov 03 07:10:28.225367 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.225347    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"
Nov 03 07:10:28.283502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.283474    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"]
Nov 03 07:10:28.330338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.330303    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225e5554-6b09-4cb8-92c4-0a0b8c02f739-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gxxn9\" (UID: \"225e5554-6b09-4cb8-92c4-0a0b8c02f739\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"
Nov 03 07:10:28.330502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.330388    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wrf6\" (UniqueName: \"kubernetes.io/projected/225e5554-6b09-4cb8-92c4-0a0b8c02f739-kube-api-access-6wrf6\") pod \"payload-pre-processing-7b98bb678c-gxxn9\" (UID: \"225e5554-6b09-4cb8-92c4-0a0b8c02f739\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"
Nov 03 07:10:28.331739 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.331718    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 07:10:28.331832 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.331746    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lztpq\""
Nov 03 07:10:28.431365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.431323    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wrf6\" (UniqueName: \"kubernetes.io/projected/225e5554-6b09-4cb8-92c4-0a0b8c02f739-kube-api-access-6wrf6\") pod \"payload-pre-processing-7b98bb678c-gxxn9\" (UID: \"225e5554-6b09-4cb8-92c4-0a0b8c02f739\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"
Nov 03 07:10:28.431727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.431705    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225e5554-6b09-4cb8-92c4-0a0b8c02f739-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gxxn9\" (UID: \"225e5554-6b09-4cb8-92c4-0a0b8c02f739\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"
Nov 03 07:10:28.432832 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.432805    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225e5554-6b09-4cb8-92c4-0a0b8c02f739-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gxxn9\" (UID: \"225e5554-6b09-4cb8-92c4-0a0b8c02f739\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"
Nov 03 07:10:28.497023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.496946    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wrf6\" (UniqueName: \"kubernetes.io/projected/225e5554-6b09-4cb8-92c4-0a0b8c02f739-kube-api-access-6wrf6\") pod \"payload-pre-processing-7b98bb678c-gxxn9\" (UID: \"225e5554-6b09-4cb8-92c4-0a0b8c02f739\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"
Nov 03 07:10:28.539544 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.539496    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"
Nov 03 07:10:28.915186 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:28.915162    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"]
Nov 03 07:10:28.918053 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:10:28.918024    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225e5554_6b09_4cb8_92c4_0a0b8c02f739.slice/crio-bc15f55de4593cfa88ab753020d9bb1e103d410dbcc54a4fd82c12955d1103f7 WatchSource:0}: Error finding container bc15f55de4593cfa88ab753020d9bb1e103d410dbcc54a4fd82c12955d1103f7: Status 404 returned error can't find the container with id bc15f55de4593cfa88ab753020d9bb1e103d410dbcc54a4fd82c12955d1103f7
Nov 03 07:10:29.564196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:29.564153    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9" event={"ID":"225e5554-6b09-4cb8-92c4-0a0b8c02f739","Type":"ContainerStarted","Data":"bc15f55de4593cfa88ab753020d9bb1e103d410dbcc54a4fd82c12955d1103f7"}
Nov 03 07:10:32.579642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:32.579602    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9" event={"ID":"225e5554-6b09-4cb8-92c4-0a0b8c02f739","Type":"ContainerStarted","Data":"1998acf7100f22264c92fe04f5987de09da9fd14dd4c1d6cd79d9a02d4771302"}
Nov 03 07:10:32.580020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:32.579715    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"
Nov 03 07:10:32.706094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:32.706042    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9" podStartSLOduration=1.970707409 podStartE2EDuration="4.706030321s" totalImagesPullingTime="2.735322912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:10:32.705409984 +0000 UTC m=+685.037778977" watchObservedRunningTime="2026-11-03 07:10:32.706030321 +0000 UTC m=+685.038399315"
Nov 03 07:10:34.758276 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:10:34.758243    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:10:36.549990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:36.549953    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:36.912673 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:36.912637    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"]
Nov 03 07:10:36.916352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:36.916333    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"
Nov 03 07:10:36.981146 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:36.981102    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"]
Nov 03 07:10:37.015312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:37.015271    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27s27\" (UniqueName: \"kubernetes.io/projected/95e7054e-d436-4502-913d-bac856b822e9-kube-api-access-27s27\") pod \"ai-gateway-controller-7f4dc97bb5-f6qq8\" (UID: \"95e7054e-d436-4502-913d-bac856b822e9\") " pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"
Nov 03 07:10:37.015312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:37.015313    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95e7054e-d436-4502-913d-bac856b822e9-metrics-certs\") pod \"ai-gateway-controller-7f4dc97bb5-f6qq8\" (UID: \"95e7054e-d436-4502-913d-bac856b822e9\") " pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"
Nov 03 07:10:37.115946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:37.115906    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27s27\" (UniqueName: \"kubernetes.io/projected/95e7054e-d436-4502-913d-bac856b822e9-kube-api-access-27s27\") pod \"ai-gateway-controller-7f4dc97bb5-f6qq8\" (UID: \"95e7054e-d436-4502-913d-bac856b822e9\") " pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"
Nov 03 07:10:37.115946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:37.115948    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95e7054e-d436-4502-913d-bac856b822e9-metrics-certs\") pod \"ai-gateway-controller-7f4dc97bb5-f6qq8\" (UID: \"95e7054e-d436-4502-913d-bac856b822e9\") " pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"
Nov 03 07:10:37.118576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:37.118548    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95e7054e-d436-4502-913d-bac856b822e9-metrics-certs\") pod \"ai-gateway-controller-7f4dc97bb5-f6qq8\" (UID: \"95e7054e-d436-4502-913d-bac856b822e9\") " pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"
Nov 03 07:10:37.181253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:37.181168    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27s27\" (UniqueName: \"kubernetes.io/projected/95e7054e-d436-4502-913d-bac856b822e9-kube-api-access-27s27\") pod \"ai-gateway-controller-7f4dc97bb5-f6qq8\" (UID: \"95e7054e-d436-4502-913d-bac856b822e9\") " pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"
Nov 03 07:10:37.228755 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:37.228721    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"
Nov 03 07:10:37.602445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:37.602412    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"]
Nov 03 07:10:37.604174 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:10:37.604150    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e7054e_d436_4502_913d_bac856b822e9.slice/crio-3f568794126a7c51938f9ab304a0dfe298d2d4dbb34c76f662e003aedda29388 WatchSource:0}: Error finding container 3f568794126a7c51938f9ab304a0dfe298d2d4dbb34c76f662e003aedda29388: Status 404 returned error can't find the container with id 3f568794126a7c51938f9ab304a0dfe298d2d4dbb34c76f662e003aedda29388
Nov 03 07:10:37.605463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:37.605447    4470 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 07:10:38.610114 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:38.610079    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8" event={"ID":"95e7054e-d436-4502-913d-bac856b822e9","Type":"ContainerStarted","Data":"206af8ee798abdf4148c48d9e0653ce9ef7facc61dbd7f8b196d4e973e507261"}
Nov 03 07:10:38.610628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:38.610164    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8" event={"ID":"95e7054e-d436-4502-913d-bac856b822e9","Type":"ContainerStarted","Data":"3f568794126a7c51938f9ab304a0dfe298d2d4dbb34c76f662e003aedda29388"}
Nov 03 07:10:38.610628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:38.610195    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"
Nov 03 07:10:38.732535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:38.732427    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8" podStartSLOduration=1.9429321370000001 podStartE2EDuration="2.732414286s" totalImagesPullingTime="789.482149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:10:38.731096807 +0000 UTC m=+691.063465804" watchObservedRunningTime="2026-11-03 07:10:38.732414286 +0000 UTC m=+691.064783280"
Nov 03 07:10:42.270586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.270555    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"]
Nov 03 07:10:42.274062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.273996    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.362232 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.362188    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.362232 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.362223    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj8qm\" (UniqueName: \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-kube-api-access-hj8qm\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.362421 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.362248    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.362421 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.362396    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.362492 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.362429    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.362492 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.362447    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.362563 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.362512    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.362648 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.362631    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.362704 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.362664    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.380909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.380882    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"]
Nov 03 07:10:42.463402 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.463368    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.463599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.463412    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.463599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.463433    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.463599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.463541    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.463735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.463603    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.463735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.463631    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.463735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.463700    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.463735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.463724    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj8qm\" (UniqueName: \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-kube-api-access-hj8qm\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.463946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.463773    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.463946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.463856    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.463946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.463909    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.464203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.464178    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.464398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.464382    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.464464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.464418    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.465770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.465749    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.466541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.466517    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.528707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.528643    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj8qm\" (UniqueName: \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-kube-api-access-hj8qm\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.529120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.529105    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gmc69\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:42.587288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:42.587255    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:43.014431 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:43.014406    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"]
Nov 03 07:10:43.016118 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:10:43.016087    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881fb74b_0ac3_4299_a577_852f5e6e6ed3.slice/crio-06a2fdbb3da3506ae37d9696fe3ef5b5a54618ca615d5702d52e47cb9e2b41b8 WatchSource:0}: Error finding container 06a2fdbb3da3506ae37d9696fe3ef5b5a54618ca615d5702d52e47cb9e2b41b8: Status 404 returned error can't find the container with id 06a2fdbb3da3506ae37d9696fe3ef5b5a54618ca615d5702d52e47cb9e2b41b8
Nov 03 07:10:43.018596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:43.018562    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:10:43.018704 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:43.018644    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:10:43.018704 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:43.018689    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:10:43.587678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:43.587646    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9"
Nov 03 07:10:43.634506 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:43.634473    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" event={"ID":"881fb74b-0ac3-4299-a577-852f5e6e6ed3","Type":"ContainerStarted","Data":"7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324"}
Nov 03 07:10:43.634663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:43.634524    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" event={"ID":"881fb74b-0ac3-4299-a577-852f5e6e6ed3","Type":"ContainerStarted","Data":"06a2fdbb3da3506ae37d9696fe3ef5b5a54618ca615d5702d52e47cb9e2b41b8"}
Nov 03 07:10:43.838605 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:43.838506    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" podStartSLOduration=1.838491738 podStartE2EDuration="1.838491738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:10:43.837646414 +0000 UTC m=+696.170015407" watchObservedRunningTime="2026-11-03 07:10:43.838491738 +0000 UTC m=+696.170860731"
Nov 03 07:10:44.588280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:44.588241    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:44.589883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:44.589857    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 07:10:44.589940 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:44.589920    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 07:10:45.587888 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:45.587850    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 07:10:45.588061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:45.587917    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 07:10:46.588121 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:46.588087    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 07:10:46.588512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:46.588169    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 07:10:47.588647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:47.588612    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 07:10:47.589047 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:47.588668    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 07:10:48.588701 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:48.588666    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 07:10:48.589097 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:48.588767    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 07:10:49.588077 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:49.588038    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 07:10:49.588270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:49.588101    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 07:10:49.615819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:49.615794    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7f4dc97bb5-f6qq8"
Nov 03 07:10:49.772004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:49.771962    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-zmd8p"]
Nov 03 07:10:49.772334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:49.772286    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p" podUID="905054cb-da87-4246-ae0a-5f02a8793111" containerName="manager" containerID="cri-o://37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a" gracePeriod=30
Nov 03 07:10:50.557878 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.557844    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:50.639460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.639428    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905054cb-da87-4246-ae0a-5f02a8793111-kube-api-access-hmnp4\" (UniqueName: \"kubernetes.io/projected/905054cb-da87-4246-ae0a-5f02a8793111-kube-api-access-hmnp4\") pod \"905054cb-da87-4246-ae0a-5f02a8793111\" (UID: \"905054cb-da87-4246-ae0a-5f02a8793111\") "
Nov 03 07:10:50.639941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.639489    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/905054cb-da87-4246-ae0a-5f02a8793111-metrics-certs\" (UniqueName: \"kubernetes.io/secret/905054cb-da87-4246-ae0a-5f02a8793111-metrics-certs\") pod \"905054cb-da87-4246-ae0a-5f02a8793111\" (UID: \"905054cb-da87-4246-ae0a-5f02a8793111\") "
Nov 03 07:10:50.641854 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.641820    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905054cb-da87-4246-ae0a-5f02a8793111-kube-api-access-hmnp4" pod "905054cb-da87-4246-ae0a-5f02a8793111" (UID: "905054cb-da87-4246-ae0a-5f02a8793111"). InnerVolumeSpecName "kube-api-access-hmnp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:10:50.641981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.641904    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/905054cb-da87-4246-ae0a-5f02a8793111-metrics-certs" pod "905054cb-da87-4246-ae0a-5f02a8793111" (UID: "905054cb-da87-4246-ae0a-5f02a8793111"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:10:50.692778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.692693    4470 generic.go:356] "Generic (PLEG): container finished" podID="905054cb-da87-4246-ae0a-5f02a8793111" containerID="37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a" exitCode=0
Nov 03 07:10:50.692778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.692754    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p"
Nov 03 07:10:50.692979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.692755    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p" event={"ID":"905054cb-da87-4246-ae0a-5f02a8793111","Type":"ContainerDied","Data":"37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a"}
Nov 03 07:10:50.693061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.693024    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-zmd8p" event={"ID":"905054cb-da87-4246-ae0a-5f02a8793111","Type":"ContainerDied","Data":"db0292c4055d7a91c5b73331a6799264107c8a87d38bb2b50e0790ac56b6f2a0"}
Nov 03 07:10:50.693118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.693064    4470 scope.go:122] "RemoveContainer" containerID="37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a"
Nov 03 07:10:50.702381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.702363    4470 scope.go:122] "RemoveContainer" containerID="37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a"
Nov 03 07:10:50.702724 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:10:50.702698    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a\": container with ID starting with 37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a not found: ID does not exist" containerID="37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a"
Nov 03 07:10:50.702813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.702738    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a"} err="failed to get container status \"37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a\": rpc error: code = NotFound desc = could not find container \"37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a\": container with ID starting with 37f0a0851d6ef83d98a4a89183e2027179c61471864fd791872f0393f12a553a not found: ID does not exist"
Nov 03 07:10:50.740645 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.740614    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmnp4\" (UniqueName: \"kubernetes.io/projected/905054cb-da87-4246-ae0a-5f02a8793111-kube-api-access-hmnp4\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:10:50.740645 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.740640    4470 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/905054cb-da87-4246-ae0a-5f02a8793111-metrics-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:10:50.865438 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.865396    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-zmd8p"]
Nov 03 07:10:50.873729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:50.873693    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-zmd8p"]
Nov 03 07:10:51.293863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:51.293825    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:51.294290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:51.294268    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:51.295085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:51.295066    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:10:51.805476 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:10:51.805445    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="905054cb-da87-4246-ae0a-5f02a8793111" path="/var/lib/kubelet/pods/905054cb-da87-4246-ae0a-5f02a8793111/volumes"
Nov 03 07:11:04.758184 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:11:04.758148    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:11:19.504189 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.504155    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"]
Nov 03 07:11:19.504612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.504554    4470 cpu_manager.go:414] "containerMap: removing container" podUID="905054cb-da87-4246-ae0a-5f02a8793111" containerName="manager"
Nov 03 07:11:19.504612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.504564    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905054cb-da87-4246-ae0a-5f02a8793111" containerName="manager"
Nov 03 07:11:19.504699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.504663    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="905054cb-da87-4246-ae0a-5f02a8793111" containerName="manager"
Nov 03 07:11:19.507243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.507224    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.562482 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.562451    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9697f7-openshift-default-dockercfg-pbdkl\""
Nov 03 07:11:19.618341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.618293    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.618530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.618358    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.618530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.618393    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.618530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.618436    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.618530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.618474    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36405c56-94e9-4818-a713-ae9a8a677975-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.618530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.618516    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m5hj\" (UniqueName: \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-kube-api-access-7m5hj\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.618793 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.618556    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.618793 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.618583    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.618793 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.618622    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36405c56-94e9-4818-a713-ae9a8a677975-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.619304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.619280    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"]
Nov 03 07:11:19.719113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719086    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.719330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719126    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.719330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719232    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.719330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719256    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.719330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719292    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36405c56-94e9-4818-a713-ae9a8a677975-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.719565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719332    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m5hj\" (UniqueName: \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-kube-api-access-7m5hj\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.719565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719373    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.719565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719401    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.719565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719440    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36405c56-94e9-4818-a713-ae9a8a677975-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.719797 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719584    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.719797 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719621    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.719906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.719875    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.720085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.720066    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.720200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.720068    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36405c56-94e9-4818-a713-ae9a8a677975-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.722328 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.722303    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.722629 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.722607    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36405c56-94e9-4818-a713-ae9a8a677975-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.783578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.783498    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.783737 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.783623    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m5hj\" (UniqueName: \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-kube-api-access-7m5hj\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:19.818567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:19.818537    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:20.206352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.206323    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"]
Nov 03 07:11:20.209625 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.209590    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.258860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.258836    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"]
Nov 03 07:11:20.260580 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:11:20.260552    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36405c56_94e9_4818_a713_ae9a8a677975.slice/crio-d791df004d5cacfa26194bd87da030427d51e9228eaaeb442f6f2b65bbf5acef WatchSource:0}: Error finding container d791df004d5cacfa26194bd87da030427d51e9228eaaeb442f6f2b65bbf5acef: Status 404 returned error can't find the container with id d791df004d5cacfa26194bd87da030427d51e9228eaaeb442f6f2b65bbf5acef
Nov 03 07:11:20.262871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.262841    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:20.262936 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.262904    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:20.262973 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.262935    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:20.265400 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.265384    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-3b41b9-openshift-default-dockercfg-zfqkd\""
Nov 03 07:11:20.308532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.308505    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"]
Nov 03 07:11:20.325323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.325297    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-678s7\" (UniqueName: \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-kube-api-access-678s7\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.325439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.325331    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.325439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.325350    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.325614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.325513    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.325614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.325549    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.325721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.325697    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.325777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.325761    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.325876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.325860    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.325942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.325928    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.427436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.427395    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.427636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.427469    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.427636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.427508    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.427636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.427569    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.427636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.427608    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-678s7\" (UniqueName: \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-kube-api-access-678s7\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.427636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.427633    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.427914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.427656    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.427914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.427711    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.427914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.427735    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.427914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.427861    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.428150 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.427944    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.428230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.428206    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.428504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.428474    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.428641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.428554    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.430124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.430102    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.430465 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.430445    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.491587 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.491509    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.491835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.491809    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-678s7\" (UniqueName: \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-kube-api-access-678s7\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.524504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.524470    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:20.535468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.535304    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"]
Nov 03 07:11:20.539538 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.539508    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.596457 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.596428    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1f2e7e-openshift-default-dockercfg-plh6k\""
Nov 03 07:11:20.611019 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.610988    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"]
Nov 03 07:11:20.731921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.731886    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.731921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.731923    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.732174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.732043    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.732174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.732089    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4219377-453a-4562-8da4-4769cd4ac358-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.732174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.732115    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4219377-453a-4562-8da4-4769cd4ac358-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.732311 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.732288    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.732385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.732367    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.732453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.732436    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmbs\" (UniqueName: \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-kube-api-access-5tmbs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.732492 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.732480    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.828557 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.828527    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" event={"ID":"36405c56-94e9-4818-a713-ae9a8a677975","Type":"ContainerStarted","Data":"0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7"}
Nov 03 07:11:20.828744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.828583    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" event={"ID":"36405c56-94e9-4818-a713-ae9a8a677975","Type":"ContainerStarted","Data":"d791df004d5cacfa26194bd87da030427d51e9228eaaeb442f6f2b65bbf5acef"}
Nov 03 07:11:20.833009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.832990    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833026    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmbs\" (UniqueName: \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-kube-api-access-5tmbs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833062    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833089    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833116    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833176    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833214    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4219377-453a-4562-8da4-4769cd4ac358-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833249    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4219377-453a-4562-8da4-4769cd4ac358-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833279    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833594 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833569    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833638    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833800 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833777    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833838    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.833923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.833895    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4219377-453a-4562-8da4-4769cd4ac358-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.835717 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.835692    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.835944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.835927    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4219377-453a-4562-8da4-4769cd4ac358-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.896748 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.896709    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.896893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.896786    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmbs\" (UniqueName: \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-kube-api-access-5tmbs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:20.918179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.916808    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"]
Nov 03 07:11:20.920874 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.920839    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:20.920951 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.920922    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:20.921007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:20.920992    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:21.005264 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.005193    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" podStartSLOduration=2.005172772 podStartE2EDuration="2.005172772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:21.002798651 +0000 UTC m=+733.335167682" watchObservedRunningTime="2026-11-03 07:11:21.005172772 +0000 UTC m=+733.337541767"
Nov 03 07:11:21.152189 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.152078    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:21.529493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.529410    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"]
Nov 03 07:11:21.531622 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:11:21.531586    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4219377_453a_4562_8da4_4769cd4ac358.slice/crio-e3b15686571a40eed29b7f0e004315afb05c9230653884f86747718758813033 WatchSource:0}: Error finding container e3b15686571a40eed29b7f0e004315afb05c9230653884f86747718758813033: Status 404 returned error can't find the container with id e3b15686571a40eed29b7f0e004315afb05c9230653884f86747718758813033
Nov 03 07:11:21.533859 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.533827    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:21.533959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.533890    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:21.533959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.533918    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:21.561476 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.561446    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"]
Nov 03 07:11:21.566116 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.566095    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.629966 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.629934    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"]
Nov 03 07:11:21.638042 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.638015    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ffee69-openshift-default-dockercfg-mdk9q\""
Nov 03 07:11:21.742745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.742713    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpmxf\" (UniqueName: \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-kube-api-access-zpmxf\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.742900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.742774    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.742900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.742886    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.743004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.742920    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.743004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.742950    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a557dbf-9da0-42bf-9354-38f46e3b9312-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.743075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.743024    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.743075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.743043    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.743175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.743094    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.743175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.743126    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.819616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.819574    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:21.827287 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.827254    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:21.837507 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.837466    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" event={"ID":"e4219377-453a-4562-8da4-4769cd4ac358","Type":"ContainerStarted","Data":"bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022"}
Nov 03 07:11:21.837693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.837539    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" event={"ID":"e4219377-453a-4562-8da4-4769cd4ac358","Type":"ContainerStarted","Data":"e3b15686571a40eed29b7f0e004315afb05c9230653884f86747718758813033"}
Nov 03 07:11:21.839176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.839148    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" event={"ID":"63c21c33-c6b9-44ee-9b22-0b1951c3d888","Type":"ContainerStarted","Data":"c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34"}
Nov 03 07:11:21.839294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.839192    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" event={"ID":"63c21c33-c6b9-44ee-9b22-0b1951c3d888","Type":"ContainerStarted","Data":"a41c0819907ae55c0c1528065e1e93181798c5e3e11ba8bade0b422aaeb5ff63"}
Nov 03 07:11:21.839493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.839477    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:21.840401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.840378    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:11:21.843773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.843755    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.843853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.843794    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpmxf\" (UniqueName: \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-kube-api-access-zpmxf\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.843853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.843835    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.843932 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.843875    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.843932 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.843903    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.844022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.843956    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a557dbf-9da0-42bf-9354-38f46e3b9312-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.844388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.844364    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.844530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.844366    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.844530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.844436    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.844530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.844472    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.844530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.844532    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.845036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.844713    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a557dbf-9da0-42bf-9354-38f46e3b9312-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.845036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.844788    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.845036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.844866    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.846481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.846458    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.846619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.846604    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.907939 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.907907    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:21.908079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:21.908027    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpmxf\" (UniqueName: \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-kube-api-access-zpmxf\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:22.152728 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.152652    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:22.157514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.157486    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:22.181812 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.181784    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:22.203053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.202996    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" podStartSLOduration=2.202977829 podStartE2EDuration="2.202977829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:22.202369889 +0000 UTC m=+734.534738882" watchObservedRunningTime="2026-11-03 07:11:22.202977829 +0000 UTC m=+734.535346824"
Nov 03 07:11:22.356971 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.356919    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" podStartSLOduration=2.35690454 podStartE2EDuration="2.35690454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:22.354614343 +0000 UTC m=+734.686983339" watchObservedRunningTime="2026-11-03 07:11:22.35690454 +0000 UTC m=+734.689273534"
Nov 03 07:11:22.525761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.525660    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:22.541757 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.541729    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:22.598615 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.598594    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"]
Nov 03 07:11:22.599961 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:11:22.599936    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a557dbf_9da0_42bf_9354_38f46e3b9312.slice/crio-41133708dcf62077bb1650374f939628fc5f9bc872558d6cc9a0037532d43fc5 WatchSource:0}: Error finding container 41133708dcf62077bb1650374f939628fc5f9bc872558d6cc9a0037532d43fc5: Status 404 returned error can't find the container with id 41133708dcf62077bb1650374f939628fc5f9bc872558d6cc9a0037532d43fc5
Nov 03 07:11:22.602250 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.602221    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:22.602298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.602284    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:22.602327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.602312    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:22.846199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.846166    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" event={"ID":"3a557dbf-9da0-42bf-9354-38f46e3b9312","Type":"ContainerStarted","Data":"624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f"}
Nov 03 07:11:22.846390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.846222    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" event={"ID":"3a557dbf-9da0-42bf-9354-38f46e3b9312","Type":"ContainerStarted","Data":"41133708dcf62077bb1650374f939628fc5f9bc872558d6cc9a0037532d43fc5"}
Nov 03 07:11:22.847001 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.846839    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:22.847001 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.846885    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:22.847444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.847427    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:22.847674 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.847660    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:11:22.974022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:22.973952    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" podStartSLOduration=1.973931694 podStartE2EDuration="1.973931694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:22.97107796 +0000 UTC m=+735.303446955" watchObservedRunningTime="2026-11-03 07:11:22.973931694 +0000 UTC m=+735.306300689"
Nov 03 07:11:23.183015 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:23.182929    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:23.187710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:23.187675    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:23.851118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:23.851080    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:23.852271 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:23.852251    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:11:34.266652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.266602    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"]
Nov 03 07:11:34.270724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.270702    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:11:34.331988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.331955    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"]
Nov 03 07:11:34.379949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.379919    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-669d6b\""
Nov 03 07:11:34.380178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.380007    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-669d6b-dockercfg-l6bll\""
Nov 03 07:11:34.432288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.432256    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"]
Nov 03 07:11:34.435805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.435783    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:11:34.463120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.463093    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27b5939-0090-4f1e-a339-472a5369058b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk\" (UID: \"f27b5939-0090-4f1e-a339-472a5369058b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:11:34.463289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.463204    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66d2h\" (UniqueName: \"kubernetes.io/projected/f27b5939-0090-4f1e-a339-472a5369058b-kube-api-access-66d2h\") pod \"payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk\" (UID: \"f27b5939-0090-4f1e-a339-472a5369058b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:11:34.469858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.469834    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"]
Nov 03 07:11:34.564659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.564634    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v\" (UID: \"3609292e-abcc-4a77-b03b-4f3d88ffa2e8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:11:34.564833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.564679    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66d2h\" (UniqueName: \"kubernetes.io/projected/f27b5939-0090-4f1e-a339-472a5369058b-kube-api-access-66d2h\") pod \"payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk\" (UID: \"f27b5939-0090-4f1e-a339-472a5369058b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:11:34.564833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.564755    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9mj\" (UniqueName: \"kubernetes.io/projected/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-kube-api-access-vl9mj\") pod \"payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v\" (UID: \"3609292e-abcc-4a77-b03b-4f3d88ffa2e8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:11:34.564833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.564823    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27b5939-0090-4f1e-a339-472a5369058b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk\" (UID: \"f27b5939-0090-4f1e-a339-472a5369058b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:11:34.565536 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.565510    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27b5939-0090-4f1e-a339-472a5369058b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk\" (UID: \"f27b5939-0090-4f1e-a339-472a5369058b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:11:34.629899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.629861    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66d2h\" (UniqueName: \"kubernetes.io/projected/f27b5939-0090-4f1e-a339-472a5369058b-kube-api-access-66d2h\") pod \"payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk\" (UID: \"f27b5939-0090-4f1e-a339-472a5369058b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:11:34.666103 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.666064    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v\" (UID: \"3609292e-abcc-4a77-b03b-4f3d88ffa2e8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:11:34.666307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.666190    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9mj\" (UniqueName: \"kubernetes.io/projected/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-kube-api-access-vl9mj\") pod \"payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v\" (UID: \"3609292e-abcc-4a77-b03b-4f3d88ffa2e8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:11:34.666757 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.666735    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v\" (UID: \"3609292e-abcc-4a77-b03b-4f3d88ffa2e8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:11:34.730370 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.730336    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9mj\" (UniqueName: \"kubernetes.io/projected/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-kube-api-access-vl9mj\") pod \"payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v\" (UID: \"3609292e-abcc-4a77-b03b-4f3d88ffa2e8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:11:34.748775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.748748    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:11:34.764975 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:11:34.764770    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:11:34.883723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:34.883650    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:11:35.129595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:35.129571    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"]
Nov 03 07:11:35.131310 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:11:35.131281    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3609292e_abcc_4a77_b03b_4f3d88ffa2e8.slice/crio-5114abadc11b6e85224a1070c05d108fb5ef4764224c4e5a05b49362ddb3215b WatchSource:0}: Error finding container 5114abadc11b6e85224a1070c05d108fb5ef4764224c4e5a05b49362ddb3215b: Status 404 returned error can't find the container with id 5114abadc11b6e85224a1070c05d108fb5ef4764224c4e5a05b49362ddb3215b
Nov 03 07:11:35.288107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:35.288080    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"]
Nov 03 07:11:35.289738 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:11:35.289711    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27b5939_0090_4f1e_a339_472a5369058b.slice/crio-2ab60e14d2aacce2199fb9c4c17d93de8c478703be6b36f1ff45cdb832f38973 WatchSource:0}: Error finding container 2ab60e14d2aacce2199fb9c4c17d93de8c478703be6b36f1ff45cdb832f38973: Status 404 returned error can't find the container with id 2ab60e14d2aacce2199fb9c4c17d93de8c478703be6b36f1ff45cdb832f38973
Nov 03 07:11:35.907693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:35.907656    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk" event={"ID":"f27b5939-0090-4f1e-a339-472a5369058b","Type":"ContainerStarted","Data":"49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1"}
Nov 03 07:11:35.907899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:35.907718    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk" event={"ID":"f27b5939-0090-4f1e-a339-472a5369058b","Type":"ContainerStarted","Data":"2ab60e14d2aacce2199fb9c4c17d93de8c478703be6b36f1ff45cdb832f38973"}
Nov 03 07:11:35.907899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:35.907829    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:11:35.909162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:35.909121    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v" event={"ID":"3609292e-abcc-4a77-b03b-4f3d88ffa2e8","Type":"ContainerStarted","Data":"577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b"}
Nov 03 07:11:35.909280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:35.909180    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v" event={"ID":"3609292e-abcc-4a77-b03b-4f3d88ffa2e8","Type":"ContainerStarted","Data":"5114abadc11b6e85224a1070c05d108fb5ef4764224c4e5a05b49362ddb3215b"}
Nov 03 07:11:35.909280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:35.909213    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:11:36.032981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.032931    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk" podStartSLOduration=2.032917178 podStartE2EDuration="2.032917178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:36.032096979 +0000 UTC m=+748.364465973" watchObservedRunningTime="2026-11-03 07:11:36.032917178 +0000 UTC m=+748.365286171"
Nov 03 07:11:36.157825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.153378    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v" podStartSLOduration=2.15335801 podStartE2EDuration="2.15335801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:36.152578338 +0000 UTC m=+748.484947364" watchObservedRunningTime="2026-11-03 07:11:36.15335801 +0000 UTC m=+748.485727007"
Nov 03 07:11:36.817091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.817058    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"]
Nov 03 07:11:36.820902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.820882    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:11:36.878288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.878258    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-1f2e7e\""
Nov 03 07:11:36.878288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.878262    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-1f2e7e-dockercfg-wvnkz\""
Nov 03 07:11:36.880230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.880209    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"]
Nov 03 07:11:36.888997 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.888971    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9n6j\" (UniqueName: \"kubernetes.io/projected/1fc99087-3fdd-4df7-9bf6-7624dc189249-kube-api-access-n9n6j\") pod \"payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq\" (UID: \"1fc99087-3fdd-4df7-9bf6-7624dc189249\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:11:36.889164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.889093    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc99087-3fdd-4df7-9bf6-7624dc189249-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq\" (UID: \"1fc99087-3fdd-4df7-9bf6-7624dc189249\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:11:36.961155 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.961107    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"]
Nov 03 07:11:36.964851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.964835    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:11:36.989801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.989773    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9n6j\" (UniqueName: \"kubernetes.io/projected/1fc99087-3fdd-4df7-9bf6-7624dc189249-kube-api-access-n9n6j\") pod \"payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq\" (UID: \"1fc99087-3fdd-4df7-9bf6-7624dc189249\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:11:36.989983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.989840    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc99087-3fdd-4df7-9bf6-7624dc189249-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq\" (UID: \"1fc99087-3fdd-4df7-9bf6-7624dc189249\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:11:36.989983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.989880    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd115615-a8ce-4efa-bc4f-5d32c851601f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2\" (UID: \"dd115615-a8ce-4efa-bc4f-5d32c851601f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:11:36.989983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.989916    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4c6\" (UniqueName: \"kubernetes.io/projected/dd115615-a8ce-4efa-bc4f-5d32c851601f-kube-api-access-7p4c6\") pod \"payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2\" (UID: \"dd115615-a8ce-4efa-bc4f-5d32c851601f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:11:36.990568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:36.990545    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc99087-3fdd-4df7-9bf6-7624dc189249-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq\" (UID: \"1fc99087-3fdd-4df7-9bf6-7624dc189249\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:11:37.025227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.025195    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"]
Nov 03 07:11:37.060609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.060581    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9n6j\" (UniqueName: \"kubernetes.io/projected/1fc99087-3fdd-4df7-9bf6-7624dc189249-kube-api-access-n9n6j\") pod \"payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq\" (UID: \"1fc99087-3fdd-4df7-9bf6-7624dc189249\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:11:37.095234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.095115    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd115615-a8ce-4efa-bc4f-5d32c851601f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2\" (UID: \"dd115615-a8ce-4efa-bc4f-5d32c851601f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:11:37.095234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.095203    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4c6\" (UniqueName: \"kubernetes.io/projected/dd115615-a8ce-4efa-bc4f-5d32c851601f-kube-api-access-7p4c6\") pod \"payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2\" (UID: \"dd115615-a8ce-4efa-bc4f-5d32c851601f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:11:37.096021 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.095997    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd115615-a8ce-4efa-bc4f-5d32c851601f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2\" (UID: \"dd115615-a8ce-4efa-bc4f-5d32c851601f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:11:37.133602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.133564    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:11:37.166375 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.166344    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4c6\" (UniqueName: \"kubernetes.io/projected/dd115615-a8ce-4efa-bc4f-5d32c851601f-kube-api-access-7p4c6\") pod \"payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2\" (UID: \"dd115615-a8ce-4efa-bc4f-5d32c851601f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:11:37.277595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.277559    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:11:37.517566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.517544    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"]
Nov 03 07:11:37.518975 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:11:37.518951    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc99087_3fdd_4df7_9bf6_7624dc189249.slice/crio-a5b02234bdf1478ff42f4446debe421b80514bb6d02de33d776995e42c4638fb WatchSource:0}: Error finding container a5b02234bdf1478ff42f4446debe421b80514bb6d02de33d776995e42c4638fb: Status 404 returned error can't find the container with id a5b02234bdf1478ff42f4446debe421b80514bb6d02de33d776995e42c4638fb
Nov 03 07:11:37.654837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.654810    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"]
Nov 03 07:11:37.655941 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:11:37.655913    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd115615_a8ce_4efa_bc4f_5d32c851601f.slice/crio-363f94c76ed0193fbe6f4a6864169719b518b796af5e7d17deb3ac89a9af68da WatchSource:0}: Error finding container 363f94c76ed0193fbe6f4a6864169719b518b796af5e7d17deb3ac89a9af68da: Status 404 returned error can't find the container with id 363f94c76ed0193fbe6f4a6864169719b518b796af5e7d17deb3ac89a9af68da
Nov 03 07:11:37.923608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.923571    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2" event={"ID":"dd115615-a8ce-4efa-bc4f-5d32c851601f","Type":"ContainerStarted","Data":"82613b64e6724240fd2bd79b11fe2521a1fe54f43d6e7af4225b40c2686ac9c1"}
Nov 03 07:11:37.924084 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.923627    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2" event={"ID":"dd115615-a8ce-4efa-bc4f-5d32c851601f","Type":"ContainerStarted","Data":"363f94c76ed0193fbe6f4a6864169719b518b796af5e7d17deb3ac89a9af68da"}
Nov 03 07:11:37.924084 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.923670    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:11:37.925334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.925309    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq" event={"ID":"1fc99087-3fdd-4df7-9bf6-7624dc189249","Type":"ContainerStarted","Data":"7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60"}
Nov 03 07:11:37.925456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.925352    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq" event={"ID":"1fc99087-3fdd-4df7-9bf6-7624dc189249","Type":"ContainerStarted","Data":"a5b02234bdf1478ff42f4446debe421b80514bb6d02de33d776995e42c4638fb"}
Nov 03 07:11:37.925456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:37.925382    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:11:38.045475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:38.045432    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2" podStartSLOduration=2.045419134 podStartE2EDuration="2.045419134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:38.043644859 +0000 UTC m=+750.376013854" watchObservedRunningTime="2026-11-03 07:11:38.045419134 +0000 UTC m=+750.377788127"
Nov 03 07:11:38.164403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:38.164348    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq" podStartSLOduration=2.164333001 podStartE2EDuration="2.164333001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:38.163550723 +0000 UTC m=+750.495919718" watchObservedRunningTime="2026-11-03 07:11:38.164333001 +0000 UTC m=+750.496702034"
Nov 03 07:11:41.965959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:41.965905    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"]
Nov 03 07:11:41.971635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:41.971608    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:11:42.025496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.025429    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"]
Nov 03 07:11:42.046491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.046450    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvzr2\" (UniqueName: \"kubernetes.io/projected/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-kube-api-access-vvzr2\") pod \"payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4\" (UID: \"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:11:42.046655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.046502    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4\" (UID: \"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:11:42.073835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.073806    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6606d6\""
Nov 03 07:11:42.073835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.073831    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6606d6-dockercfg-hdqjw\""
Nov 03 07:11:42.147649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.147616    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4\" (UID: \"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:11:42.147820 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.147769    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvzr2\" (UniqueName: \"kubernetes.io/projected/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-kube-api-access-vvzr2\") pod \"payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4\" (UID: \"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:11:42.148334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.148318    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4\" (UID: \"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:11:42.217271 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.217209    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvzr2\" (UniqueName: \"kubernetes.io/projected/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-kube-api-access-vvzr2\") pod \"payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4\" (UID: \"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:11:42.286039 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.286009    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:11:42.664981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.664939    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"]
Nov 03 07:11:42.666569 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:11:42.666542    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d2c70b_2ef0_45c2_9bce_8a02d58db1c9.slice/crio-86fe6ece3eee9c7ffc77cbc706df6b445be7b789df5bebe225dc7d28ab7d4ba0 WatchSource:0}: Error finding container 86fe6ece3eee9c7ffc77cbc706df6b445be7b789df5bebe225dc7d28ab7d4ba0: Status 404 returned error can't find the container with id 86fe6ece3eee9c7ffc77cbc706df6b445be7b789df5bebe225dc7d28ab7d4ba0
Nov 03 07:11:42.954723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.954627    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4" event={"ID":"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9","Type":"ContainerStarted","Data":"e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4"}
Nov 03 07:11:42.954723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.954679    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4" event={"ID":"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9","Type":"ContainerStarted","Data":"86fe6ece3eee9c7ffc77cbc706df6b445be7b789df5bebe225dc7d28ab7d4ba0"}
Nov 03 07:11:42.954950 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:42.954773    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:11:43.075484 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:43.075433    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4" podStartSLOduration=2.075418391 podStartE2EDuration="2.075418391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:43.075306235 +0000 UTC m=+755.407675228" watchObservedRunningTime="2026-11-03 07:11:43.075418391 +0000 UTC m=+755.407787386"
Nov 03 07:11:46.920826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:46.920794    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:11:46.922257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:46.922238    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:11:46.946089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:46.945760    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"]
Nov 03 07:11:46.954170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:46.950881    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:11:47.008755 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.008705    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbm2\" (UniqueName: \"kubernetes.io/projected/dd717e60-b059-4bd2-9ca5-386a081b0f5d-kube-api-access-5tbm2\") pod \"payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp\" (UID: \"dd717e60-b059-4bd2-9ca5-386a081b0f5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:11:47.008927 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.008878    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd717e60-b059-4bd2-9ca5-386a081b0f5d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp\" (UID: \"dd717e60-b059-4bd2-9ca5-386a081b0f5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:11:47.011612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.011589    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"]
Nov 03 07:11:47.027406 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.027376    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ffee69-dockercfg-bhn7b\""
Nov 03 07:11:47.027584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.027390    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ffee69\""
Nov 03 07:11:47.109925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.109885    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbm2\" (UniqueName: \"kubernetes.io/projected/dd717e60-b059-4bd2-9ca5-386a081b0f5d-kube-api-access-5tbm2\") pod \"payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp\" (UID: \"dd717e60-b059-4bd2-9ca5-386a081b0f5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:11:47.110083 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.110031    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd717e60-b059-4bd2-9ca5-386a081b0f5d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp\" (UID: \"dd717e60-b059-4bd2-9ca5-386a081b0f5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:11:47.110784 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.110761    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd717e60-b059-4bd2-9ca5-386a081b0f5d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp\" (UID: \"dd717e60-b059-4bd2-9ca5-386a081b0f5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:11:47.178231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.178071    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbm2\" (UniqueName: \"kubernetes.io/projected/dd717e60-b059-4bd2-9ca5-386a081b0f5d-kube-api-access-5tbm2\") pod \"payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp\" (UID: \"dd717e60-b059-4bd2-9ca5-386a081b0f5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:11:47.269469 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.269430    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:11:47.649512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.649487    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"]
Nov 03 07:11:47.650992 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:11:47.650960    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd717e60_b059_4bd2_9ca5_386a081b0f5d.slice/crio-e79f5dece03abaeae210127e51245b3ea1d2b08236ca08622cbdda29ba5cf49a WatchSource:0}: Error finding container e79f5dece03abaeae210127e51245b3ea1d2b08236ca08622cbdda29ba5cf49a: Status 404 returned error can't find the container with id e79f5dece03abaeae210127e51245b3ea1d2b08236ca08622cbdda29ba5cf49a
Nov 03 07:11:47.979656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.979568    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp" event={"ID":"dd717e60-b059-4bd2-9ca5-386a081b0f5d","Type":"ContainerStarted","Data":"5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944"}
Nov 03 07:11:47.979656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.979616    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp" event={"ID":"dd717e60-b059-4bd2-9ca5-386a081b0f5d","Type":"ContainerStarted","Data":"e79f5dece03abaeae210127e51245b3ea1d2b08236ca08622cbdda29ba5cf49a"}
Nov 03 07:11:47.980050 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:47.979669    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:11:48.117435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:48.117383    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp" podStartSLOduration=2.117368076 podStartE2EDuration="2.117368076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:48.11541684 +0000 UTC m=+760.447785833" watchObservedRunningTime="2026-11-03 07:11:48.117368076 +0000 UTC m=+760.449737070"
Nov 03 07:11:48.933404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:48.933336    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:11:48.933404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:48.933396    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:11:49.506498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.506464    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"]
Nov 03 07:11:49.510860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.510842    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:11:49.569160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.568492    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-3fea3a-dockercfg-g5hjx\""
Nov 03 07:11:49.569372 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.569182    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-3fea3a\""
Nov 03 07:11:49.621485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.621453    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"]
Nov 03 07:11:49.643810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.643272    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f47047-32fd-45de-9f31-f1df5fb713da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r\" (UID: \"66f47047-32fd-45de-9f31-f1df5fb713da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:11:49.643810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.643318    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7254m\" (UniqueName: \"kubernetes.io/projected/66f47047-32fd-45de-9f31-f1df5fb713da-kube-api-access-7254m\") pod \"payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r\" (UID: \"66f47047-32fd-45de-9f31-f1df5fb713da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:11:49.705160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.704278    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"]
Nov 03 07:11:49.710004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.709965    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:11:49.726314 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.726282    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"]
Nov 03 07:11:49.744191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.744164    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f47047-32fd-45de-9f31-f1df5fb713da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r\" (UID: \"66f47047-32fd-45de-9f31-f1df5fb713da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:11:49.744339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.744216    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7254m\" (UniqueName: \"kubernetes.io/projected/66f47047-32fd-45de-9f31-f1df5fb713da-kube-api-access-7254m\") pod \"payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r\" (UID: \"66f47047-32fd-45de-9f31-f1df5fb713da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:11:49.744904 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.744885    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f47047-32fd-45de-9f31-f1df5fb713da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r\" (UID: \"66f47047-32fd-45de-9f31-f1df5fb713da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:11:49.810261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.810053    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7254m\" (UniqueName: \"kubernetes.io/projected/66f47047-32fd-45de-9f31-f1df5fb713da-kube-api-access-7254m\") pod \"payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r\" (UID: \"66f47047-32fd-45de-9f31-f1df5fb713da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:11:49.823573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.823548    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:11:49.845342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.845284    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjpz\" (UniqueName: \"kubernetes.io/projected/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-kube-api-access-xdjpz\") pod \"payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l\" (UID: \"c710b3c4-f1e8-4a84-bd87-a02c252b7e09\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:11:49.880758 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.845488    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l\" (UID: \"c710b3c4-f1e8-4a84-bd87-a02c252b7e09\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:11:49.962513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.950199    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjpz\" (UniqueName: \"kubernetes.io/projected/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-kube-api-access-xdjpz\") pod \"payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l\" (UID: \"c710b3c4-f1e8-4a84-bd87-a02c252b7e09\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:11:49.962513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.958327    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l\" (UID: \"c710b3c4-f1e8-4a84-bd87-a02c252b7e09\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:11:49.968391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:49.968333    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l\" (UID: \"c710b3c4-f1e8-4a84-bd87-a02c252b7e09\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:11:50.014608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:50.014580    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjpz\" (UniqueName: \"kubernetes.io/projected/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-kube-api-access-xdjpz\") pod \"payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l\" (UID: \"c710b3c4-f1e8-4a84-bd87-a02c252b7e09\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:11:50.024453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:50.024420    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:11:50.299161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:50.298209    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"]
Nov 03 07:11:50.304732 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:11:50.304607    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f47047_32fd_45de_9f31_f1df5fb713da.slice/crio-8f279c28943f533f29c786d9a8843f897a17fe08eaf6aa744dffb8852953ed90 WatchSource:0}: Error finding container 8f279c28943f533f29c786d9a8843f897a17fe08eaf6aa744dffb8852953ed90: Status 404 returned error can't find the container with id 8f279c28943f533f29c786d9a8843f897a17fe08eaf6aa744dffb8852953ed90
Nov 03 07:11:50.487156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:50.469046    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"]
Nov 03 07:11:51.008499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:51.008267    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l" event={"ID":"c710b3c4-f1e8-4a84-bd87-a02c252b7e09","Type":"ContainerStarted","Data":"dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78"}
Nov 03 07:11:51.008499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:51.008320    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l" event={"ID":"c710b3c4-f1e8-4a84-bd87-a02c252b7e09","Type":"ContainerStarted","Data":"b4282ecba618b18ce7ee0a9bedc728a0ccf9658ded1e6a76bfbbe188b0b5c569"}
Nov 03 07:11:51.008499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:51.008341    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:11:51.010110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:51.010086    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r" event={"ID":"66f47047-32fd-45de-9f31-f1df5fb713da","Type":"ContainerStarted","Data":"d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d"}
Nov 03 07:11:51.010247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:51.010149    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:11:51.010247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:51.010167    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r" event={"ID":"66f47047-32fd-45de-9f31-f1df5fb713da","Type":"ContainerStarted","Data":"8f279c28943f533f29c786d9a8843f897a17fe08eaf6aa744dffb8852953ed90"}
Nov 03 07:11:51.137102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:51.137042    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l" podStartSLOduration=2.137021984 podStartE2EDuration="2.137021984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:51.13489071 +0000 UTC m=+763.467259704" watchObservedRunningTime="2026-11-03 07:11:51.137021984 +0000 UTC m=+763.469390980"
Nov 03 07:11:51.268637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:51.268508    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r" podStartSLOduration=2.268486634 podStartE2EDuration="2.268486634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:51.265564663 +0000 UTC m=+763.597933659" watchObservedRunningTime="2026-11-03 07:11:51.268486634 +0000 UTC m=+763.600855635"
Nov 03 07:11:52.225868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.225836    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"]
Nov 03 07:11:52.229902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.229884    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:11:52.298840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.298807    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"]
Nov 03 07:11:52.323920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.323887    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5zj\" (UniqueName: \"kubernetes.io/projected/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-kube-api-access-wr5zj\") pod \"payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6\" (UID: \"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:11:52.323920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.323922    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6\" (UID: \"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:11:52.339173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.336240    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-3b41b9\""
Nov 03 07:11:52.339173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.336328    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-3b41b9-dockercfg-b5wqc\""
Nov 03 07:11:52.426206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.426166    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5zj\" (UniqueName: \"kubernetes.io/projected/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-kube-api-access-wr5zj\") pod \"payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6\" (UID: \"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:11:52.426389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.426219    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6\" (UID: \"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:11:52.427020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.426995    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6\" (UID: \"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:11:52.509558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.509479    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5zj\" (UniqueName: \"kubernetes.io/projected/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-kube-api-access-wr5zj\") pod \"payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6\" (UID: \"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:11:52.543302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:52.543272    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:11:53.004165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:53.002303    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"]
Nov 03 07:11:53.008301 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:11:53.008267    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a71dad7_b3b4_4b7b_b69b_c81585ef7ab7.slice/crio-4b3af277fbe3a8e8672b99841372b3180b93c1a9e142113ff75f2c2e38929773 WatchSource:0}: Error finding container 4b3af277fbe3a8e8672b99841372b3180b93c1a9e142113ff75f2c2e38929773: Status 404 returned error can't find the container with id 4b3af277fbe3a8e8672b99841372b3180b93c1a9e142113ff75f2c2e38929773
Nov 03 07:11:53.022116 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:53.022083    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6" event={"ID":"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7","Type":"ContainerStarted","Data":"4b3af277fbe3a8e8672b99841372b3180b93c1a9e142113ff75f2c2e38929773"}
Nov 03 07:11:53.983158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:53.980304    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:11:54.028241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:54.028211    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6" event={"ID":"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7","Type":"ContainerStarted","Data":"92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767"}
Nov 03 07:11:54.028785 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:54.028769    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:11:54.230841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:54.230792    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6" podStartSLOduration=2.230777469 podStartE2EDuration="2.230777469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:54.230088699 +0000 UTC m=+766.562457693" watchObservedRunningTime="2026-11-03 07:11:54.230777469 +0000 UTC m=+766.563146463"
Nov 03 07:11:58.056068 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:58.056030    4470 generic.go:356] "Generic (PLEG): container finished" podID="e4219377-453a-4562-8da4-4769cd4ac358" containerID="bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022" exitCode=0
Nov 03 07:11:58.057631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:58.056072    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" event={"ID":"e4219377-453a-4562-8da4-4769cd4ac358","Type":"ContainerDied","Data":"bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022"}
Nov 03 07:11:58.057631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:58.056567    4470 scope.go:122] "RemoveContainer" containerID="bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022"
Nov 03 07:11:58.058208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:58.058033    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:58.058208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:58.058115    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:58.058208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:58.058196    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:58.987059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:58.987032    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:11:59.075839 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:59.075808    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" event={"ID":"e4219377-453a-4562-8da4-4769cd4ac358","Type":"ContainerStarted","Data":"d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c"}
Nov 03 07:11:59.077449 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:59.077427    4470 generic.go:356] "Generic (PLEG): container finished" podID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerID="624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f" exitCode=0
Nov 03 07:11:59.077565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:59.077501    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" event={"ID":"3a557dbf-9da0-42bf-9354-38f46e3b9312","Type":"ContainerDied","Data":"624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f"}
Nov 03 07:11:59.077859 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:59.077845    4470 scope.go:122] "RemoveContainer" containerID="624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f"
Nov 03 07:11:59.079074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:59.079046    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:59.079172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:59.079109    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:59.079172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:59.079158    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:11:59.171656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:59.171609    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:11:59.171975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:59.171905    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 07:11:59.171975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:11:59.171949    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 07:12:00.094041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:00.093727    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" event={"ID":"3a557dbf-9da0-42bf-9354-38f46e3b9312","Type":"ContainerStarted","Data":"9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af"}
Nov 03 07:12:00.162158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:00.161275    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 07:12:00.162158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:00.161349    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 07:12:00.185165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:00.184525    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:12:00.185165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:00.184644    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 07:12:00.185165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:00.184685    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 07:12:01.171158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:01.168441    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 07:12:01.171158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:01.168499    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 07:12:01.189170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:01.188575    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 07:12:01.189170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:01.188637    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 07:12:02.018520 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:02.018494    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:12:02.018725 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:02.018561    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:12:02.155601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:02.155560    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 07:12:02.155887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:02.155853    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 07:12:02.187163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:02.185975    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 07:12:02.187163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:02.186125    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 07:12:03.164160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:03.155591    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 07:12:03.164503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:03.164460    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 07:12:03.186579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:03.186534    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 07:12:03.186838 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:03.186811    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 07:12:04.154587 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:04.154549    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 07:12:04.155209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:04.155179    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 07:12:04.182592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:04.182554    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 07:12:04.182776 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:04.182616    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 07:12:04.783032 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:12:04.782994    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:12:05.161390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:05.161356    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:12:05.161837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:05.161733    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:12:05.162439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:05.162418    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:12:05.182598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:05.182562    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 07:12:05.182731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:05.182630    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 07:12:06.045958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:06.045931    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:12:06.187318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:06.187283    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:12:06.187770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:06.187723    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:12:06.188446 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:06.188426    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:12:17.208207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:17.208174    4470 generic.go:356] "Generic (PLEG): container finished" podID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerID="c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34" exitCode=0
Nov 03 07:12:17.208661 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:17.208253    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" event={"ID":"63c21c33-c6b9-44ee-9b22-0b1951c3d888","Type":"ContainerDied","Data":"c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34"}
Nov 03 07:12:17.208703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:17.208682    4470 scope.go:122] "RemoveContainer" containerID="c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34"
Nov 03 07:12:17.209995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:17.209963    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:12:17.210099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:17.210075    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:12:17.210180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:17.210127    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:12:18.215707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:18.215671    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" event={"ID":"63c21c33-c6b9-44ee-9b22-0b1951c3d888","Type":"ContainerStarted","Data":"da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2"}
Nov 03 07:12:18.525270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:18.525176    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:12:18.526491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:18.526468    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 07:12:18.526676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:18.526654    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 07:12:19.525495 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:19.525462    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 07:12:19.525992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:19.525962    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 07:12:20.525777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:20.525741    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 07:12:20.526208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:20.525791    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 07:12:21.525766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:21.525730    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 07:12:21.525944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:21.525789    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 07:12:22.525095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:22.525063    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 07:12:22.525290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:22.525113    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 07:12:22.857286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:22.857245    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:12:23.584781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:23.584747    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:12:23.587283 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:23.587260    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:12:26.281831 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:26.281802    4470 generic.go:356] "Generic (PLEG): container finished" podID="36405c56-94e9-4818-a713-ae9a8a677975" containerID="0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7" exitCode=0
Nov 03 07:12:26.282278 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:26.281881    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" event={"ID":"36405c56-94e9-4818-a713-ae9a8a677975","Type":"ContainerDied","Data":"0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7"}
Nov 03 07:12:26.282321 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:26.282300    4470 scope.go:122] "RemoveContainer" containerID="0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7"
Nov 03 07:12:26.283598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:26.283573    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:12:26.283655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:26.283638    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:12:26.283687 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:26.283669    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:12:27.296219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:27.296188    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" event={"ID":"36405c56-94e9-4818-a713-ae9a8a677975","Type":"ContainerStarted","Data":"3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f"}
Nov 03 07:12:27.819298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:27.819267    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:12:27.819501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:27.819480    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 07:12:27.819556 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:27.819542    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 07:12:28.819568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:28.819528    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 07:12:28.819968 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:28.819586    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 07:12:29.818920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:29.818884    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 07:12:29.819113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:29.818947    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 07:12:30.819224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:30.819192    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 07:12:30.819626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:30.819280    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 07:12:31.819438 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:31.819400    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 07:12:31.819920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:31.819454    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 07:12:32.823912 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:32.823878    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:12:32.824333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:32.824167    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:12:32.824773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:12:32.824758    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:12:34.759095 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:12:34.759064    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:13:04.759059 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:13:04.759021    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:13:17.454209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.454175    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"]
Nov 03 07:13:17.458653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.458631    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.504848 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.504814    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-model-cache\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.505020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.504964    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-home\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.505020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.504997    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf4c497a-71a5-4cf8-866f-7630b8961efa-tls-certs\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.505172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.505087    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-dshm\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.505172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.505111    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfgxh\" (UniqueName: \"kubernetes.io/projected/bf4c497a-71a5-4cf8-866f-7630b8961efa-kube-api-access-pfgxh\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.505172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.505159    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-tmp-dir\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.560719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.560688    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3fea3a\"/\"openshift-service-ca.crt\""
Nov 03 07:13:17.564199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.564171    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"]
Nov 03 07:13:17.566609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.566590    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3fea3a\"/\"default-dockercfg-7vn7m\""
Nov 03 07:13:17.566709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.566618    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3fea3a\"/\"kube-root-ca.crt\""
Nov 03 07:13:17.566709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.566618    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3fea3a\"/\"simulator-w5-3fea3a-kserve-self-signed-certs\""
Nov 03 07:13:17.605923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.605874    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-home\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.606122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.605931    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf4c497a-71a5-4cf8-866f-7630b8961efa-tls-certs\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.606122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.606022    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-dshm\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.606122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.606047    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgxh\" (UniqueName: \"kubernetes.io/projected/bf4c497a-71a5-4cf8-866f-7630b8961efa-kube-api-access-pfgxh\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.606122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.606079    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-tmp-dir\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.606959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.606201    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-model-cache\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.606959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.606657    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-model-cache\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.606959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.606846    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-home\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.607120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.607014    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-tmp-dir\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.609422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.609394    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-dshm\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.609575 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.609559    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf4c497a-71a5-4cf8-866f-7630b8961efa-tls-certs\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.671037 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.671003    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgxh\" (UniqueName: \"kubernetes.io/projected/bf4c497a-71a5-4cf8-866f-7630b8961efa-kube-api-access-pfgxh\") pod \"simulator-w5-3fea3a-kserve-56d768978f-68gs2\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") " pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:17.804591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:17.804507    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:18.183970 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:18.183950    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"]
Nov 03 07:13:18.185625 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:13:18.185602    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf4c497a_71a5_4cf8_866f_7630b8961efa.slice/crio-6059e1e08cf57255294c1ed8c380c63ff1ed0d5279885f8cd3cbfdcd4f5995c5 WatchSource:0}: Error finding container 6059e1e08cf57255294c1ed8c380c63ff1ed0d5279885f8cd3cbfdcd4f5995c5: Status 404 returned error can't find the container with id 6059e1e08cf57255294c1ed8c380c63ff1ed0d5279885f8cd3cbfdcd4f5995c5
Nov 03 07:13:18.586114 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:18.586082    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" event={"ID":"bf4c497a-71a5-4cf8-866f-7630b8961efa","Type":"ContainerStarted","Data":"340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372"}
Nov 03 07:13:18.586537 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:18.586124    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" event={"ID":"bf4c497a-71a5-4cf8-866f-7630b8961efa","Type":"ContainerStarted","Data":"6059e1e08cf57255294c1ed8c380c63ff1ed0d5279885f8cd3cbfdcd4f5995c5"}
Nov 03 07:13:18.709480 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:18.709426    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podStartSLOduration=1.7094104639999999 podStartE2EDuration="1.709410464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:13:18.707505798 +0000 UTC m=+851.039874793" watchObservedRunningTime="2026-11-03 07:13:18.709410464 +0000 UTC m=+851.041779457"
Nov 03 07:13:18.805460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:18.805430    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:27.811901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:27.811868    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:27.819044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:27.819015    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:28.657345 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:28.657319    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:13:34.758422 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:13:34.758384    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:13:37.763502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.763471    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"]
Nov 03 07:13:37.767311 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.767296    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:37.826617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.826588    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"]
Nov 03 07:13:37.874670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.874638    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"default-dockercfg-td9rn\""
Nov 03 07:13:37.874821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.874709    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"kube-root-ca.crt\""
Nov 03 07:13:37.874872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.874817    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"premium-w2-669d6b-kserve-self-signed-certs\""
Nov 03 07:13:37.874938 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.874924    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"openshift-service-ca.crt\""
Nov 03 07:13:37.926041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.926012    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-model-cache\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:37.926234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.926055    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tls-certs\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:37.926234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.926225    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjqn9\" (UniqueName: \"kubernetes.io/projected/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-kube-api-access-rjqn9\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:37.926309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.926250    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-home\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:37.926309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.926285    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-dshm\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:37.926393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:37.926377    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tmp-dir\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.027432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.027363    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjqn9\" (UniqueName: \"kubernetes.io/projected/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-kube-api-access-rjqn9\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.027432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.027396    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-home\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.027660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.027438    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-dshm\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.027660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.027614    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tmp-dir\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.027848 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.027710    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-model-cache\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.027848 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.027746    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tls-certs\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.027954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.027930    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-home\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.028035 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.028003    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tmp-dir\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.028112 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.028083    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-model-cache\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.029799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.029784    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-dshm\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.030316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.030297    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tls-certs\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.095565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.095538    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjqn9\" (UniqueName: \"kubernetes.io/projected/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-kube-api-access-rjqn9\") pod \"premium-w2-669d6b-kserve-6f8bd456d7-q9hp2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") " pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.380094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.380064    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:38.758190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:38.758168    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"]
Nov 03 07:13:38.759671 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:13:38.759643    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ddd1670_b8a3_4687_8f43_2918ad60b7a2.slice/crio-841f110adcaf5342e0ae0975b25c3de7dde19085d6252d5fc1f9b37e62d9ea6c WatchSource:0}: Error finding container 841f110adcaf5342e0ae0975b25c3de7dde19085d6252d5fc1f9b37e62d9ea6c: Status 404 returned error can't find the container with id 841f110adcaf5342e0ae0975b25c3de7dde19085d6252d5fc1f9b37e62d9ea6c
Nov 03 07:13:39.705851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:39.705815    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" event={"ID":"7ddd1670-b8a3-4687-8f43-2918ad60b7a2","Type":"ContainerStarted","Data":"e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4"}
Nov 03 07:13:39.706359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:39.705858    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" event={"ID":"7ddd1670-b8a3-4687-8f43-2918ad60b7a2","Type":"ContainerStarted","Data":"841f110adcaf5342e0ae0975b25c3de7dde19085d6252d5fc1f9b37e62d9ea6c"}
Nov 03 07:13:39.826821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:39.826777    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podStartSLOduration=2.826764599 podStartE2EDuration="2.826764599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:13:39.82663903 +0000 UTC m=+872.159008024" watchObservedRunningTime="2026-11-03 07:13:39.826764599 +0000 UTC m=+872.159133594"
Nov 03 07:13:40.380329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:40.380297    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:48.380460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:48.380426    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:48.393223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:48.393194    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:13:48.764539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:13:48.764460    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:14:02.210188 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.210156    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"]
Nov 03 07:14:02.213809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.213791    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.278200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.278168    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"]
Nov 03 07:14:02.288713 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.288687    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tmp-dir\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.288881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.288725    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tls-certs\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.288881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.288763    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-dshm\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.288961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.288904    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-model-cache\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.288961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.288937    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-home\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.289050 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.288965    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbqzt\" (UniqueName: \"kubernetes.io/projected/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-kube-api-access-bbqzt\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.321570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.321539    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ffee69\"/\"openshift-service-ca.crt\""
Nov 03 07:14:02.321762 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.321599    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ffee69\"/\"default-dockercfg-r7hhd\""
Nov 03 07:14:02.321762 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.321634    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ffee69\"/\"premium-w4-ffee69-kserve-self-signed-certs\""
Nov 03 07:14:02.321896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.321878    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ffee69\"/\"kube-root-ca.crt\""
Nov 03 07:14:02.389869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.389840    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbqzt\" (UniqueName: \"kubernetes.io/projected/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-kube-api-access-bbqzt\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.390076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.389905    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tmp-dir\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.390076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.389932    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tls-certs\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.390076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.389969    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-dshm\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.390076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.390031    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-model-cache\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.390076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.390069    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-home\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.390430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.390390    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tmp-dir\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.390474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.390436    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-home\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.390527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.390504    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-model-cache\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.392481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.392455    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-dshm\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.392583 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.392568    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tls-certs\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.455200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.455167    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbqzt\" (UniqueName: \"kubernetes.io/projected/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-kube-api-access-bbqzt\") pod \"premium-w4-ffee69-kserve-6487b8fd79-9sc6f\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") " pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.526760 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.526678    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:02.917479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:02.917454    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"]
Nov 03 07:14:02.918941 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:14:02.918914    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0db08b8e_2684_4a15_8eee_2bd8d07c9ee4.slice/crio-88be6709f740ff8fdc4049ef4b9f6336a2e5354e74862092649aa34e4a237dae WatchSource:0}: Error finding container 88be6709f740ff8fdc4049ef4b9f6336a2e5354e74862092649aa34e4a237dae: Status 404 returned error can't find the container with id 88be6709f740ff8fdc4049ef4b9f6336a2e5354e74862092649aa34e4a237dae
Nov 03 07:14:03.833457 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:03.833424    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" event={"ID":"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4","Type":"ContainerStarted","Data":"f51019aa959d32b857d7062165266b1ac175caf25f86e8eb60f103544f26de23"}
Nov 03 07:14:03.833457 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:03.833464    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" event={"ID":"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4","Type":"ContainerStarted","Data":"88be6709f740ff8fdc4049ef4b9f6336a2e5354e74862092649aa34e4a237dae"}
Nov 03 07:14:03.958914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:03.958860    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podStartSLOduration=1.95884751 podStartE2EDuration="1.95884751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:14:03.95834836 +0000 UTC m=+896.290717354" watchObservedRunningTime="2026-11-03 07:14:03.95884751 +0000 UTC m=+896.291216503"
Nov 03 07:14:04.526952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:04.526911    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:04.760279 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:14:04.760242    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:14:12.527673 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:12.527641    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:12.540014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:12.539990    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:12.902088 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:12.902061    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:14:34.758522 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:14:34.758484    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:14:41.783765 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:41.783729    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"]
Nov 03 07:14:41.787658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:41.787641    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:41.895458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:41.895423    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3fea3a\"/\"premium-w5-3fea3a-kserve-self-signed-certs\""
Nov 03 07:14:41.897693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:41.897670    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"]
Nov 03 07:14:41.909176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:41.909120    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-dshm\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:41.909327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:41.909201    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-model-cache\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:41.909327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:41.909287    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-home\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:41.909447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:41.909342    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tmp-dir\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:41.909574 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:41.909558    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llsfx\" (UniqueName: \"kubernetes.io/projected/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-kube-api-access-llsfx\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:41.909634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:41.909585    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tls-certs\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.010787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.010746    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llsfx\" (UniqueName: \"kubernetes.io/projected/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-kube-api-access-llsfx\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.010787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.010790    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tls-certs\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.011061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.010836    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-dshm\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.011061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.010853    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-model-cache\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.011061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.010873    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-home\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.011061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.010897    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tmp-dir\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.011346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.011327    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-model-cache\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.011422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.011398    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-home\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.011459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.011425    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tmp-dir\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.013417 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.013389    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-dshm\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.013690 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.013671    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tls-certs\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.076872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.076839    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llsfx\" (UniqueName: \"kubernetes.io/projected/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-kube-api-access-llsfx\") pod \"premium-w5-3fea3a-kserve-568f4956b8-rc6wq\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") " pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.099044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.099008    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:42.480299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:42.480272    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"]
Nov 03 07:14:42.482052 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:14:42.482028    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode293d8c4_9555_4b8e_bae6_75b88e66a2f4.slice/crio-44242c0537dab4f9259d3b8546a475e46a3239cf120c3082f673e631692ab178 WatchSource:0}: Error finding container 44242c0537dab4f9259d3b8546a475e46a3239cf120c3082f673e631692ab178: Status 404 returned error can't find the container with id 44242c0537dab4f9259d3b8546a475e46a3239cf120c3082f673e631692ab178
Nov 03 07:14:43.059472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:43.059439    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" event={"ID":"e293d8c4-9555-4b8e-bae6-75b88e66a2f4","Type":"ContainerStarted","Data":"f9074f12e5b66412c6d04e939ee1e425929bb2093b4322bcd7a8f0014b52b654"}
Nov 03 07:14:43.059472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:43.059476    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" event={"ID":"e293d8c4-9555-4b8e-bae6-75b88e66a2f4","Type":"ContainerStarted","Data":"44242c0537dab4f9259d3b8546a475e46a3239cf120c3082f673e631692ab178"}
Nov 03 07:14:43.099768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:43.099741    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:43.188518 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:43.188459    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podStartSLOduration=2.188446073 podStartE2EDuration="2.188446073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:14:43.18557739 +0000 UTC m=+935.517946383" watchObservedRunningTime="2026-11-03 07:14:43.188446073 +0000 UTC m=+935.520815067"
Nov 03 07:14:52.099583 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:52.099496    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:52.112378 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:52.112354    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:52.126845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:52.126820    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:14:56.215056 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.215019    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"]
Nov 03 07:14:56.218811 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.218796    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:14:56.264537 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.264509    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5fe16c8-5kffr\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:14:56.264744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.264576    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4f8\" (UniqueName: \"kubernetes.io/projected/ae3fd299-28ea-4746-b665-95dd3c823ebe-kube-api-access-rg4f8\") pod \"maas-api-revoke-sub-c5fe16c8-5kffr\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:14:56.264744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.264714    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5fe16c8-5kffr\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:14:56.277778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.277753    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"]
Nov 03 07:14:56.325252 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.325228    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:14:56.325423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.325396    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:14:56.325423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.325403    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:14:56.325520 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.325464    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:14:56.365487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.365457    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5fe16c8-5kffr\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:14:56.365616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.365520    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5fe16c8-5kffr\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:14:56.365616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.365561    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4f8\" (UniqueName: \"kubernetes.io/projected/ae3fd299-28ea-4746-b665-95dd3c823ebe-kube-api-access-rg4f8\") pod \"maas-api-revoke-sub-c5fe16c8-5kffr\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:14:56.366190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.366169    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5fe16c8-5kffr\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:14:56.368231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.368212    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5fe16c8-5kffr\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:14:56.430318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.430287    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4f8\" (UniqueName: \"kubernetes.io/projected/ae3fd299-28ea-4746-b665-95dd3c823ebe-kube-api-access-rg4f8\") pod \"maas-api-revoke-sub-c5fe16c8-5kffr\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:14:56.532004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.531930    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:14:56.908497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:56.908453    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"]
Nov 03 07:14:56.909660 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:14:56.909638    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3fd299_28ea_4746_b665_95dd3c823ebe.slice/crio-7488c489fe3dd1d2ae96c0e576266d299cd5952f9784f6bd6798cef2ab798ef0 WatchSource:0}: Error finding container 7488c489fe3dd1d2ae96c0e576266d299cd5952f9784f6bd6798cef2ab798ef0: Status 404 returned error can't find the container with id 7488c489fe3dd1d2ae96c0e576266d299cd5952f9784f6bd6798cef2ab798ef0
Nov 03 07:14:57.144883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:14:57.144852    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr" event={"ID":"ae3fd299-28ea-4746-b665-95dd3c823ebe","Type":"ContainerStarted","Data":"7488c489fe3dd1d2ae96c0e576266d299cd5952f9784f6bd6798cef2ab798ef0"}
Nov 03 07:15:00.179374 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.179333    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"]
Nov 03 07:15:00.188380 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.188346    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"]
Nov 03 07:15:00.188542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.188460    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"
Nov 03 07:15:00.192974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.192950    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"]
Nov 03 07:15:00.193124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.193056    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"
Nov 03 07:15:00.196950 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.196928    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"
Nov 03 07:15:00.199106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.199083    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h\" (UID: \"e57cbb60-e47b-4b77-a68c-5f09ba6ce906\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"
Nov 03 07:15:00.199225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.199213    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqs6s\" (UniqueName: \"kubernetes.io/projected/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-kube-api-access-wqs6s\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h\" (UID: \"e57cbb60-e47b-4b77-a68c-5f09ba6ce906\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"
Nov 03 07:15:00.199306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.199288    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxgct\" (UniqueName: \"kubernetes.io/projected/cffac00c-a46c-4016-86b9-80ecfc91228e-kube-api-access-lxgct\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs\" (UID: \"cffac00c-a46c-4016-86b9-80ecfc91228e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"
Nov 03 07:15:00.199429 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.199411    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cffac00c-a46c-4016-86b9-80ecfc91228e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs\" (UID: \"cffac00c-a46c-4016-86b9-80ecfc91228e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"
Nov 03 07:15:00.199499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.199456    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57m26\" (UniqueName: \"kubernetes.io/projected/003bacd2-9437-4842-b8c6-8abfc15ec54e-kube-api-access-57m26\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66\" (UID: \"003bacd2-9437-4842-b8c6-8abfc15ec54e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"
Nov 03 07:15:00.199499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.199489    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003bacd2-9437-4842-b8c6-8abfc15ec54e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66\" (UID: \"003bacd2-9437-4842-b8c6-8abfc15ec54e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"
Nov 03 07:15:00.228497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.228467    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"]
Nov 03 07:15:00.233056 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.233035    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"]
Nov 03 07:15:00.233182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.233123    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"
Nov 03 07:15:00.237906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.237887    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"
Nov 03 07:15:00.281605 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.281570    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"]
Nov 03 07:15:00.294463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.294180    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"]
Nov 03 07:15:00.294463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.294257    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"]
Nov 03 07:15:00.295566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.295543    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"]
Nov 03 07:15:00.300306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.300195    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxgct\" (UniqueName: \"kubernetes.io/projected/cffac00c-a46c-4016-86b9-80ecfc91228e-kube-api-access-lxgct\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs\" (UID: \"cffac00c-a46c-4016-86b9-80ecfc91228e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"
Nov 03 07:15:00.300306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.300300    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cffac00c-a46c-4016-86b9-80ecfc91228e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs\" (UID: \"cffac00c-a46c-4016-86b9-80ecfc91228e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"
Nov 03 07:15:00.300799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.300335    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fds\" (UniqueName: \"kubernetes.io/projected/28eb79bc-36bf-4ccc-ac22-e2d66597c572-kube-api-access-j9fds\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm\" (UID: \"28eb79bc-36bf-4ccc-ac22-e2d66597c572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"
Nov 03 07:15:00.300799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.300374    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57m26\" (UniqueName: \"kubernetes.io/projected/003bacd2-9437-4842-b8c6-8abfc15ec54e-kube-api-access-57m26\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66\" (UID: \"003bacd2-9437-4842-b8c6-8abfc15ec54e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"
Nov 03 07:15:00.300799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.300402    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003bacd2-9437-4842-b8c6-8abfc15ec54e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66\" (UID: \"003bacd2-9437-4842-b8c6-8abfc15ec54e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"
Nov 03 07:15:00.300799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.300427    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs\" (UID: \"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"
Nov 03 07:15:00.300799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.300477    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28eb79bc-36bf-4ccc-ac22-e2d66597c572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm\" (UID: \"28eb79bc-36bf-4ccc-ac22-e2d66597c572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"
Nov 03 07:15:00.300799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.300514    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h\" (UID: \"e57cbb60-e47b-4b77-a68c-5f09ba6ce906\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"
Nov 03 07:15:00.300799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.300554    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbtqq\" (UniqueName: \"kubernetes.io/projected/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-kube-api-access-wbtqq\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs\" (UID: \"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"
Nov 03 07:15:00.300799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.300735    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqs6s\" (UniqueName: \"kubernetes.io/projected/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-kube-api-access-wqs6s\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h\" (UID: \"e57cbb60-e47b-4b77-a68c-5f09ba6ce906\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"
Nov 03 07:15:00.301358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.301300    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003bacd2-9437-4842-b8c6-8abfc15ec54e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66\" (UID: \"003bacd2-9437-4842-b8c6-8abfc15ec54e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"
Nov 03 07:15:00.301358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.301318    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h\" (UID: \"e57cbb60-e47b-4b77-a68c-5f09ba6ce906\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"
Nov 03 07:15:00.302026 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.301963    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cffac00c-a46c-4016-86b9-80ecfc91228e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs\" (UID: \"cffac00c-a46c-4016-86b9-80ecfc91228e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"
Nov 03 07:15:00.335426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.334846    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"]
Nov 03 07:15:00.364961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.364895    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqs6s\" (UniqueName: \"kubernetes.io/projected/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-kube-api-access-wqs6s\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h\" (UID: \"e57cbb60-e47b-4b77-a68c-5f09ba6ce906\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"
Nov 03 07:15:00.364961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.364912    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxgct\" (UniqueName: \"kubernetes.io/projected/cffac00c-a46c-4016-86b9-80ecfc91228e-kube-api-access-lxgct\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs\" (UID: \"cffac00c-a46c-4016-86b9-80ecfc91228e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"
Nov 03 07:15:00.365447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.365426    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57m26\" (UniqueName: \"kubernetes.io/projected/003bacd2-9437-4842-b8c6-8abfc15ec54e-kube-api-access-57m26\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66\" (UID: \"003bacd2-9437-4842-b8c6-8abfc15ec54e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"
Nov 03 07:15:00.401867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.401830    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs\" (UID: \"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"
Nov 03 07:15:00.402075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.401917    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28eb79bc-36bf-4ccc-ac22-e2d66597c572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm\" (UID: \"28eb79bc-36bf-4ccc-ac22-e2d66597c572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"
Nov 03 07:15:00.402075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.401964    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtqq\" (UniqueName: \"kubernetes.io/projected/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-kube-api-access-wbtqq\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs\" (UID: \"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"
Nov 03 07:15:00.402244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.402116    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fds\" (UniqueName: \"kubernetes.io/projected/28eb79bc-36bf-4ccc-ac22-e2d66597c572-kube-api-access-j9fds\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm\" (UID: \"28eb79bc-36bf-4ccc-ac22-e2d66597c572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"
Nov 03 07:15:00.402586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.402558    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs\" (UID: \"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"
Nov 03 07:15:00.402675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.402614    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28eb79bc-36bf-4ccc-ac22-e2d66597c572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm\" (UID: \"28eb79bc-36bf-4ccc-ac22-e2d66597c572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"
Nov 03 07:15:00.467737 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.467668    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtqq\" (UniqueName: \"kubernetes.io/projected/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-kube-api-access-wbtqq\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs\" (UID: \"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"
Nov 03 07:15:00.469059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.469018    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fds\" (UniqueName: \"kubernetes.io/projected/28eb79bc-36bf-4ccc-ac22-e2d66597c572-kube-api-access-j9fds\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm\" (UID: \"28eb79bc-36bf-4ccc-ac22-e2d66597c572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"
Nov 03 07:15:00.508083 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.508045    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"
Nov 03 07:15:00.519464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.519419    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"
Nov 03 07:15:00.526193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.526125    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"
Nov 03 07:15:00.549268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.549240    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"
Nov 03 07:15:00.557465 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.557433    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"
Nov 03 07:15:00.909037 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.909007    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"]
Nov 03 07:15:00.921966 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:15:00.921942    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003bacd2_9437_4842_b8c6_8abfc15ec54e.slice/crio-c45f1a8e00af909e2cbf74b3147f2a992c4876feeadb18b0d289681617165606 WatchSource:0}: Error finding container c45f1a8e00af909e2cbf74b3147f2a992c4876feeadb18b0d289681617165606: Status 404 returned error can't find the container with id c45f1a8e00af909e2cbf74b3147f2a992c4876feeadb18b0d289681617165606
Nov 03 07:15:00.959656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.959618    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"]
Nov 03 07:15:00.960381 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:15:00.959942    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffac00c_a46c_4016_86b9_80ecfc91228e.slice/crio-174f79e0e4f4f5844d09a2c63960a66bda352793fd84b1e785e033a616f7110a WatchSource:0}: Error finding container 174f79e0e4f4f5844d09a2c63960a66bda352793fd84b1e785e033a616f7110a: Status 404 returned error can't find the container with id 174f79e0e4f4f5844d09a2c63960a66bda352793fd84b1e785e033a616f7110a
Nov 03 07:15:00.962421 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.962400    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"]
Nov 03 07:15:00.974174 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:15:00.974124    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57cbb60_e47b_4b77_a68c_5f09ba6ce906.slice/crio-4a6c48a16274cc6d30845118502df6bb4bc1d09454a12e3f00df433dfd7623fb WatchSource:0}: Error finding container 4a6c48a16274cc6d30845118502df6bb4bc1d09454a12e3f00df433dfd7623fb: Status 404 returned error can't find the container with id 4a6c48a16274cc6d30845118502df6bb4bc1d09454a12e3f00df433dfd7623fb
Nov 03 07:15:00.990845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:00.990734    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"]
Nov 03 07:15:01.005348 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:15:01.005323    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8442daaf_3bdd_4b9f_9d54_e8ad9a1eb0f2.slice/crio-5cb4a1cee33f50b9c89eccb621624ddb705acdde0f69661c4bc801021b11eb12 WatchSource:0}: Error finding container 5cb4a1cee33f50b9c89eccb621624ddb705acdde0f69661c4bc801021b11eb12: Status 404 returned error can't find the container with id 5cb4a1cee33f50b9c89eccb621624ddb705acdde0f69661c4bc801021b11eb12
Nov 03 07:15:01.017359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:01.016531    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"]
Nov 03 07:15:01.168899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:01.168870    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs" event={"ID":"cffac00c-a46c-4016-86b9-80ecfc91228e","Type":"ContainerStarted","Data":"174f79e0e4f4f5844d09a2c63960a66bda352793fd84b1e785e033a616f7110a"}
Nov 03 07:15:01.170036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:01.170015    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs" event={"ID":"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2","Type":"ContainerStarted","Data":"5cb4a1cee33f50b9c89eccb621624ddb705acdde0f69661c4bc801021b11eb12"}
Nov 03 07:15:01.171073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:01.171054    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66" event={"ID":"003bacd2-9437-4842-b8c6-8abfc15ec54e","Type":"ContainerStarted","Data":"c45f1a8e00af909e2cbf74b3147f2a992c4876feeadb18b0d289681617165606"}
Nov 03 07:15:01.172177 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:01.172156    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm" event={"ID":"28eb79bc-36bf-4ccc-ac22-e2d66597c572","Type":"ContainerStarted","Data":"4bd8b0b1ef3fae4bbaf8fd3dfc5c5d8d68821983804611e9b007eff8602704e4"}
Nov 03 07:15:01.174634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:01.174614    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h" event={"ID":"e57cbb60-e47b-4b77-a68c-5f09ba6ce906","Type":"ContainerStarted","Data":"4a6c48a16274cc6d30845118502df6bb4bc1d09454a12e3f00df433dfd7623fb"}
Nov 03 07:15:01.294667 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:01.294612    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr" podStartSLOduration=1.45310969 podStartE2EDuration="5.294597111s" totalImagesPullingTime="3.841487421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:01.293686279 +0000 UTC m=+953.626055275" watchObservedRunningTime="2026-11-03 07:15:01.294597111 +0000 UTC m=+953.626966105"
Nov 03 07:15:02.182817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.182780    4470 generic.go:356] "Generic (PLEG): container finished" podID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerID="1a99a2f1cf98c6c7ec4395edc1684e0e44f33976105a93b39dbf0563332c9696" exitCode=60
Nov 03 07:15:02.183003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.182865    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs" event={"ID":"cffac00c-a46c-4016-86b9-80ecfc91228e","Type":"ContainerDied","Data":"1a99a2f1cf98c6c7ec4395edc1684e0e44f33976105a93b39dbf0563332c9696"}
Nov 03 07:15:02.183178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.183159    4470 scope.go:122] "RemoveContainer" containerID="1a99a2f1cf98c6c7ec4395edc1684e0e44f33976105a93b39dbf0563332c9696"
Nov 03 07:15:02.184923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.184896    4470 generic.go:356] "Generic (PLEG): container finished" podID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerID="deb055e821fa2f14d540e999dfdf613dc27e38297d4e6db7c223fa3036820bea" exitCode=60
Nov 03 07:15:02.185021 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.184981    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs" event={"ID":"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2","Type":"ContainerDied","Data":"deb055e821fa2f14d540e999dfdf613dc27e38297d4e6db7c223fa3036820bea"}
Nov 03 07:15:02.185289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.185270    4470 scope.go:122] "RemoveContainer" containerID="deb055e821fa2f14d540e999dfdf613dc27e38297d4e6db7c223fa3036820bea"
Nov 03 07:15:02.187384 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.187360    4470 generic.go:356] "Generic (PLEG): container finished" podID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerID="b1e08ded6c110d6d5138346c743f1aad905ddc20066acbb125e7b76dc8860462" exitCode=60
Nov 03 07:15:02.187473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.187450    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66" event={"ID":"003bacd2-9437-4842-b8c6-8abfc15ec54e","Type":"ContainerDied","Data":"b1e08ded6c110d6d5138346c743f1aad905ddc20066acbb125e7b76dc8860462"}
Nov 03 07:15:02.187707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.187671    4470 scope.go:122] "RemoveContainer" containerID="b1e08ded6c110d6d5138346c743f1aad905ddc20066acbb125e7b76dc8860462"
Nov 03 07:15:02.189484 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.189410    4470 generic.go:356] "Generic (PLEG): container finished" podID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerID="53d219061f3e5b0950663e5d2d97b2cc9cb215fdf075b1274f11a7749e8da54b" exitCode=60
Nov 03 07:15:02.189543 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.189524    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm" event={"ID":"28eb79bc-36bf-4ccc-ac22-e2d66597c572","Type":"ContainerDied","Data":"53d219061f3e5b0950663e5d2d97b2cc9cb215fdf075b1274f11a7749e8da54b"}
Nov 03 07:15:02.189827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.189754    4470 scope.go:122] "RemoveContainer" containerID="53d219061f3e5b0950663e5d2d97b2cc9cb215fdf075b1274f11a7749e8da54b"
Nov 03 07:15:02.191282 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.191262    4470 generic.go:356] "Generic (PLEG): container finished" podID="ae3fd299-28ea-4746-b665-95dd3c823ebe" containerID="3f37625371d91f33cbfe38241389cb22f887ba8224981d47ee75b69a39815918" exitCode=0
Nov 03 07:15:02.191362 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.191331    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr" event={"ID":"ae3fd299-28ea-4746-b665-95dd3c823ebe","Type":"ContainerDied","Data":"3f37625371d91f33cbfe38241389cb22f887ba8224981d47ee75b69a39815918"}
Nov 03 07:15:02.192896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.192879    4470 generic.go:356] "Generic (PLEG): container finished" podID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerID="3fc3c807add016882bf3779e565524b3a6d9d7ea9ca83909938fc84dff7b3fe0" exitCode=60
Nov 03 07:15:02.192979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.192922    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h" event={"ID":"e57cbb60-e47b-4b77-a68c-5f09ba6ce906","Type":"ContainerDied","Data":"3fc3c807add016882bf3779e565524b3a6d9d7ea9ca83909938fc84dff7b3fe0"}
Nov 03 07:15:02.193173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:02.193159    4470 scope.go:122] "RemoveContainer" containerID="3fc3c807add016882bf3779e565524b3a6d9d7ea9ca83909938fc84dff7b3fe0"
Nov 03 07:15:03.200564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.200528    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66" event={"ID":"003bacd2-9437-4842-b8c6-8abfc15ec54e","Type":"ContainerStarted","Data":"9ca75ff54d9e973ca7949a9425a7aca8453de15b037aeb794cef6f56a7709f8c"}
Nov 03 07:15:03.202581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.202554    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm" event={"ID":"28eb79bc-36bf-4ccc-ac22-e2d66597c572","Type":"ContainerStarted","Data":"f2f6a26a43c4915ba94e766ec56a38128a8cef17e4e742b07b66ff5302c87669"}
Nov 03 07:15:03.204342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.204321    4470 generic.go:356] "Generic (PLEG): container finished" podID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerID="fc8e67941015bae58fb4450a08eb9cf925ec4a9c4ab0f52119f210bf95a1b36a" exitCode=60
Nov 03 07:15:03.204463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.204400    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h" event={"ID":"e57cbb60-e47b-4b77-a68c-5f09ba6ce906","Type":"ContainerDied","Data":"fc8e67941015bae58fb4450a08eb9cf925ec4a9c4ab0f52119f210bf95a1b36a"}
Nov 03 07:15:03.204463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.204456    4470 scope.go:122] "RemoveContainer" containerID="3fc3c807add016882bf3779e565524b3a6d9d7ea9ca83909938fc84dff7b3fe0"
Nov 03 07:15:03.204704 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.204689    4470 scope.go:122] "RemoveContainer" containerID="fc8e67941015bae58fb4450a08eb9cf925ec4a9c4ab0f52119f210bf95a1b36a"
Nov 03 07:15:03.204961 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:03.204937    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h_odh-ai-gateway-infra(e57cbb60-e47b-4b77-a68c-5f09ba6ce906)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906"
Nov 03 07:15:03.206348 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.206324    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs" event={"ID":"cffac00c-a46c-4016-86b9-80ecfc91228e","Type":"ContainerStarted","Data":"3f295449ea978715213ae874da4a3a99583e9c8665a1e24f2186ff914d09b03b"}
Nov 03 07:15:03.208056 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.208038    4470 generic.go:356] "Generic (PLEG): container finished" podID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerID="8964266129f231dcf5ac76ede5f1cf45ac3dd8395304c5ef88ab813c98882621" exitCode=60
Nov 03 07:15:03.208128 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.208060    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs" event={"ID":"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2","Type":"ContainerDied","Data":"8964266129f231dcf5ac76ede5f1cf45ac3dd8395304c5ef88ab813c98882621"}
Nov 03 07:15:03.208355 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.208341    4470 scope.go:122] "RemoveContainer" containerID="8964266129f231dcf5ac76ede5f1cf45ac3dd8395304c5ef88ab813c98882621"
Nov 03 07:15:03.208583 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:03.208565    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs_odh-ai-gateway-infra(8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2"
Nov 03 07:15:03.323316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.323271    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66" podStartSLOduration=3.323255259 podStartE2EDuration="3.323255259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:03.320880363 +0000 UTC m=+955.653249362" watchObservedRunningTime="2026-11-03 07:15:03.323255259 +0000 UTC m=+955.655624253"
Nov 03 07:15:03.452719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.452687    4470 scope.go:122] "RemoveContainer" containerID="deb055e821fa2f14d540e999dfdf613dc27e38297d4e6db7c223fa3036820bea"
Nov 03 07:15:03.459557 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.459533    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:15:03.536602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.536563    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-openshift-ca\") pod \"ae3fd299-28ea-4746-b665-95dd3c823ebe\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") "
Nov 03 07:15:03.536754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.536619    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3fd299-28ea-4746-b665-95dd3c823ebe-kube-api-access-rg4f8\" (UniqueName: \"kubernetes.io/projected/ae3fd299-28ea-4746-b665-95dd3c823ebe-kube-api-access-rg4f8\") pod \"ae3fd299-28ea-4746-b665-95dd3c823ebe\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") "
Nov 03 07:15:03.536813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.536782    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-xks-ca\") pod \"ae3fd299-28ea-4746-b665-95dd3c823ebe\" (UID: \"ae3fd299-28ea-4746-b665-95dd3c823ebe\") "
Nov 03 07:15:03.536916 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.536894    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-openshift-ca" pod "ae3fd299-28ea-4746-b665-95dd3c823ebe" (UID: "ae3fd299-28ea-4746-b665-95dd3c823ebe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:15:03.537106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.537084    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:03.539145 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.539102    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-xks-ca" pod "ae3fd299-28ea-4746-b665-95dd3c823ebe" (UID: "ae3fd299-28ea-4746-b665-95dd3c823ebe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:15:03.539261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.539158    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3fd299-28ea-4746-b665-95dd3c823ebe-kube-api-access-rg4f8" pod "ae3fd299-28ea-4746-b665-95dd3c823ebe" (UID: "ae3fd299-28ea-4746-b665-95dd3c823ebe"). InnerVolumeSpecName "kube-api-access-rg4f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:15:03.561097 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.561044    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm" podStartSLOduration=3.5610300539999997 podStartE2EDuration="3.561030054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:03.559024387 +0000 UTC m=+955.891393381" watchObservedRunningTime="2026-11-03 07:15:03.561030054 +0000 UTC m=+955.893399048"
Nov 03 07:15:03.638147 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.638110    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3fd299-28ea-4746-b665-95dd3c823ebe-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:03.638319 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.638174    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg4f8\" (UniqueName: \"kubernetes.io/projected/ae3fd299-28ea-4746-b665-95dd3c823ebe-kube-api-access-rg4f8\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:03.800079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:03.799959    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs" podStartSLOduration=3.7999425860000002 podStartE2EDuration="3.799942586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:03.798676151 +0000 UTC m=+956.131045146" watchObservedRunningTime="2026-11-03 07:15:03.799942586 +0000 UTC m=+956.132311581"
Nov 03 07:15:04.215824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.215788    4470 scope.go:122] "RemoveContainer" containerID="fc8e67941015bae58fb4450a08eb9cf925ec4a9c4ab0f52119f210bf95a1b36a"
Nov 03 07:15:04.216332 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:04.216069    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h_odh-ai-gateway-infra(e57cbb60-e47b-4b77-a68c-5f09ba6ce906)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906"
Nov 03 07:15:04.217317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.217291    4470 generic.go:356] "Generic (PLEG): container finished" podID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerID="3f295449ea978715213ae874da4a3a99583e9c8665a1e24f2186ff914d09b03b" exitCode=60
Nov 03 07:15:04.217437 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.217413    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs" event={"ID":"cffac00c-a46c-4016-86b9-80ecfc91228e","Type":"ContainerDied","Data":"3f295449ea978715213ae874da4a3a99583e9c8665a1e24f2186ff914d09b03b"}
Nov 03 07:15:04.217493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.217477    4470 scope.go:122] "RemoveContainer" containerID="1a99a2f1cf98c6c7ec4395edc1684e0e44f33976105a93b39dbf0563332c9696"
Nov 03 07:15:04.217597 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.217584    4470 scope.go:122] "RemoveContainer" containerID="3f295449ea978715213ae874da4a3a99583e9c8665a1e24f2186ff914d09b03b"
Nov 03 07:15:04.217847 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:04.217829    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs_odh-ai-gateway-infra(cffac00c-a46c-4016-86b9-80ecfc91228e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e"
Nov 03 07:15:04.219184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.219166    4470 scope.go:122] "RemoveContainer" containerID="8964266129f231dcf5ac76ede5f1cf45ac3dd8395304c5ef88ab813c98882621"
Nov 03 07:15:04.219419 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:04.219396    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs_odh-ai-gateway-infra(8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2"
Nov 03 07:15:04.220681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.220663    4470 generic.go:356] "Generic (PLEG): container finished" podID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerID="9ca75ff54d9e973ca7949a9425a7aca8453de15b037aeb794cef6f56a7709f8c" exitCode=60
Nov 03 07:15:04.220771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.220715    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66" event={"ID":"003bacd2-9437-4842-b8c6-8abfc15ec54e","Type":"ContainerDied","Data":"9ca75ff54d9e973ca7949a9425a7aca8453de15b037aeb794cef6f56a7709f8c"}
Nov 03 07:15:04.220893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.220881    4470 scope.go:122] "RemoveContainer" containerID="9ca75ff54d9e973ca7949a9425a7aca8453de15b037aeb794cef6f56a7709f8c"
Nov 03 07:15:04.221062 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:04.221046    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66_odh-ai-gateway-infra(003bacd2-9437-4842-b8c6-8abfc15ec54e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e"
Nov 03 07:15:04.222254 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.222236    4470 generic.go:356] "Generic (PLEG): container finished" podID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerID="f2f6a26a43c4915ba94e766ec56a38128a8cef17e4e742b07b66ff5302c87669" exitCode=60
Nov 03 07:15:04.222326 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.222314    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm" event={"ID":"28eb79bc-36bf-4ccc-ac22-e2d66597c572","Type":"ContainerDied","Data":"f2f6a26a43c4915ba94e766ec56a38128a8cef17e4e742b07b66ff5302c87669"}
Nov 03 07:15:04.222481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.222464    4470 scope.go:122] "RemoveContainer" containerID="f2f6a26a43c4915ba94e766ec56a38128a8cef17e4e742b07b66ff5302c87669"
Nov 03 07:15:04.222689 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:04.222671    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm_odh-ai-gateway-infra(28eb79bc-36bf-4ccc-ac22-e2d66597c572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572"
Nov 03 07:15:04.223619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.223600    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr" event={"ID":"ae3fd299-28ea-4746-b665-95dd3c823ebe","Type":"ContainerDied","Data":"7488c489fe3dd1d2ae96c0e576266d299cd5952f9784f6bd6798cef2ab798ef0"}
Nov 03 07:15:04.223619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.223611    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"
Nov 03 07:15:04.223760 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.223631    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7488c489fe3dd1d2ae96c0e576266d299cd5952f9784f6bd6798cef2ab798ef0"
Nov 03 07:15:04.230609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.230592    4470 scope.go:122] "RemoveContainer" containerID="b1e08ded6c110d6d5138346c743f1aad905ddc20066acbb125e7b76dc8860462"
Nov 03 07:15:04.242419 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:04.242403    4470 scope.go:122] "RemoveContainer" containerID="53d219061f3e5b0950663e5d2d97b2cc9cb215fdf075b1274f11a7749e8da54b"
Nov 03 07:15:04.760260 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:04.760222    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:15:05.230977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:05.230951    4470 scope.go:122] "RemoveContainer" containerID="9ca75ff54d9e973ca7949a9425a7aca8453de15b037aeb794cef6f56a7709f8c"
Nov 03 07:15:05.231458 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:05.231333    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66_odh-ai-gateway-infra(003bacd2-9437-4842-b8c6-8abfc15ec54e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e"
Nov 03 07:15:05.232675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:05.232641    4470 scope.go:122] "RemoveContainer" containerID="f2f6a26a43c4915ba94e766ec56a38128a8cef17e4e742b07b66ff5302c87669"
Nov 03 07:15:05.232927 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:05.232902    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm_odh-ai-gateway-infra(28eb79bc-36bf-4ccc-ac22-e2d66597c572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572"
Nov 03 07:15:05.234198 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:05.234183    4470 scope.go:122] "RemoveContainer" containerID="3f295449ea978715213ae874da4a3a99583e9c8665a1e24f2186ff914d09b03b"
Nov 03 07:15:05.234376 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:05.234359    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs_odh-ai-gateway-infra(cffac00c-a46c-4016-86b9-80ecfc91228e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e"
Nov 03 07:15:11.863424 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:11.863392    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"]
Nov 03 07:15:11.863952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:11.863939    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ae3fd299-28ea-4746-b665-95dd3c823ebe" containerName="revoke-keys"
Nov 03 07:15:11.864000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:11.863954    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3fd299-28ea-4746-b665-95dd3c823ebe" containerName="revoke-keys"
Nov 03 07:15:11.864039 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:11.864031    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3fd299-28ea-4746-b665-95dd3c823ebe" containerName="revoke-keys"
Nov 03 07:15:11.868761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:11.868747    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:11.924378 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:11.924350    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:15:11.931935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:11.931909    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d853984-bpz2f\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:11.932124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:11.932100    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d853984-bpz2f\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:11.932290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:11.932272    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdtd4\" (UniqueName: \"kubernetes.io/projected/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-kube-api-access-fdtd4\") pod \"maas-api-revoke-sub-3d853984-bpz2f\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:11.966700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:11.966665    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"]
Nov 03 07:15:12.033191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:12.033125    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdtd4\" (UniqueName: \"kubernetes.io/projected/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-kube-api-access-fdtd4\") pod \"maas-api-revoke-sub-3d853984-bpz2f\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:12.033395 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:12.033242    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d853984-bpz2f\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:12.033395 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:12.033299    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d853984-bpz2f\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:12.033988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:12.033968    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d853984-bpz2f\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:12.035824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:12.035805    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d853984-bpz2f\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:12.097592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:12.097559    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdtd4\" (UniqueName: \"kubernetes.io/projected/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-kube-api-access-fdtd4\") pod \"maas-api-revoke-sub-3d853984-bpz2f\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:12.182286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:12.182216    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:12.557561 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:12.557529    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"]
Nov 03 07:15:12.559184 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:15:12.559126    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95072d50_f6f0_431a_b7e5_395a7c3dc2d7.slice/crio-5eec3eebcb716d0a1b6657955eb50eb30982c48042464c1bd9bf8be48616fb95 WatchSource:0}: Error finding container 5eec3eebcb716d0a1b6657955eb50eb30982c48042464c1bd9bf8be48616fb95: Status 404 returned error can't find the container with id 5eec3eebcb716d0a1b6657955eb50eb30982c48042464c1bd9bf8be48616fb95
Nov 03 07:15:13.291614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:13.291518    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f" event={"ID":"95072d50-f6f0-431a-b7e5-395a7c3dc2d7","Type":"ContainerStarted","Data":"707a7b321dc2f50aa51bb9c73dd6408d1159e46639ca9245ff11d6d27905e3e7"}
Nov 03 07:15:13.291614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:13.291578    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f" event={"ID":"95072d50-f6f0-431a-b7e5-395a7c3dc2d7","Type":"ContainerStarted","Data":"5eec3eebcb716d0a1b6657955eb50eb30982c48042464c1bd9bf8be48616fb95"}
Nov 03 07:15:13.414050 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:13.414001    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f" podStartSLOduration=2.413987826 podStartE2EDuration="2.413987826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:13.411646186 +0000 UTC m=+965.744015180" watchObservedRunningTime="2026-11-03 07:15:13.413987826 +0000 UTC m=+965.746356856"
Nov 03 07:15:13.799804 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:13.799772    4470 scope.go:122] "RemoveContainer" containerID="3f295449ea978715213ae874da4a3a99583e9c8665a1e24f2186ff914d09b03b"
Nov 03 07:15:14.299580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:14.299544    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs" event={"ID":"cffac00c-a46c-4016-86b9-80ecfc91228e","Type":"ContainerStarted","Data":"fb1cb62d18fcdf84a0723c49c05462bde9c5a349c0217b9ef5ff300850f6669c"}
Nov 03 07:15:14.301068 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:14.301043    4470 generic.go:356] "Generic (PLEG): container finished" podID="95072d50-f6f0-431a-b7e5-395a7c3dc2d7" containerID="707a7b321dc2f50aa51bb9c73dd6408d1159e46639ca9245ff11d6d27905e3e7" exitCode=0
Nov 03 07:15:14.301226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:14.301119    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f" event={"ID":"95072d50-f6f0-431a-b7e5-395a7c3dc2d7","Type":"ContainerDied","Data":"707a7b321dc2f50aa51bb9c73dd6408d1159e46639ca9245ff11d6d27905e3e7"}
Nov 03 07:15:14.800219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:14.800191    4470 scope.go:122] "RemoveContainer" containerID="fc8e67941015bae58fb4450a08eb9cf925ec4a9c4ab0f52119f210bf95a1b36a"
Nov 03 07:15:14.800356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:14.800306    4470 scope.go:122] "RemoveContainer" containerID="9ca75ff54d9e973ca7949a9425a7aca8453de15b037aeb794cef6f56a7709f8c"
Nov 03 07:15:15.309156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.309109    4470 generic.go:356] "Generic (PLEG): container finished" podID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerID="fb1cb62d18fcdf84a0723c49c05462bde9c5a349c0217b9ef5ff300850f6669c" exitCode=60
Nov 03 07:15:15.309633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.309180    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs" event={"ID":"cffac00c-a46c-4016-86b9-80ecfc91228e","Type":"ContainerDied","Data":"fb1cb62d18fcdf84a0723c49c05462bde9c5a349c0217b9ef5ff300850f6669c"}
Nov 03 07:15:15.309633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.309239    4470 scope.go:122] "RemoveContainer" containerID="3f295449ea978715213ae874da4a3a99583e9c8665a1e24f2186ff914d09b03b"
Nov 03 07:15:15.309633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.309579    4470 scope.go:122] "RemoveContainer" containerID="fb1cb62d18fcdf84a0723c49c05462bde9c5a349c0217b9ef5ff300850f6669c"
Nov 03 07:15:15.309882 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:15.309863    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs_odh-ai-gateway-infra(cffac00c-a46c-4016-86b9-80ecfc91228e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e"
Nov 03 07:15:15.311284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.311266    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66" event={"ID":"003bacd2-9437-4842-b8c6-8abfc15ec54e","Type":"ContainerStarted","Data":"47e9bc48b85f22364438b0ff1682295f8c3dd3f195fe6c4956046abd0580ef26"}
Nov 03 07:15:15.312955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.312933    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h" event={"ID":"e57cbb60-e47b-4b77-a68c-5f09ba6ce906","Type":"ContainerStarted","Data":"c065829c1d3df962dce1cbb0020c894dc1fe1a2426769680d43fd3b37967819b"}
Nov 03 07:15:15.442479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.442457    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"]
Nov 03 07:15:15.458729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.458704    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:15.573633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.573558    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-openshift-ca\") pod \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") "
Nov 03 07:15:15.573773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.573649    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-kube-api-access-fdtd4\" (UniqueName: \"kubernetes.io/projected/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-kube-api-access-fdtd4\") pod \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") "
Nov 03 07:15:15.573773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.573726    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-xks-ca\") pod \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\" (UID: \"95072d50-f6f0-431a-b7e5-395a7c3dc2d7\") "
Nov 03 07:15:15.573968 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.573942    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-openshift-ca" pod "95072d50-f6f0-431a-b7e5-395a7c3dc2d7" (UID: "95072d50-f6f0-431a-b7e5-395a7c3dc2d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:15:15.574110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.574087    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:15.575830 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.575811    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-xks-ca" pod "95072d50-f6f0-431a-b7e5-395a7c3dc2d7" (UID: "95072d50-f6f0-431a-b7e5-395a7c3dc2d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:15:15.575982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.575963    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-kube-api-access-fdtd4" pod "95072d50-f6f0-431a-b7e5-395a7c3dc2d7" (UID: "95072d50-f6f0-431a-b7e5-395a7c3dc2d7"). InnerVolumeSpecName "kube-api-access-fdtd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:15:15.670535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.670486    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h" podStartSLOduration=15.670472995 podStartE2EDuration="15.670472995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:15.669518758 +0000 UTC m=+968.001887754" watchObservedRunningTime="2026-11-03 07:15:15.670472995 +0000 UTC m=+968.002841988"
Nov 03 07:15:15.675088 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.675069    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:15.675171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:15.675091    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdtd4\" (UniqueName: \"kubernetes.io/projected/95072d50-f6f0-431a-b7e5-395a7c3dc2d7-kube-api-access-fdtd4\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:16.320219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.320187    4470 generic.go:356] "Generic (PLEG): container finished" podID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerID="47e9bc48b85f22364438b0ff1682295f8c3dd3f195fe6c4956046abd0580ef26" exitCode=60
Nov 03 07:15:16.320636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.320266    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66" event={"ID":"003bacd2-9437-4842-b8c6-8abfc15ec54e","Type":"ContainerDied","Data":"47e9bc48b85f22364438b0ff1682295f8c3dd3f195fe6c4956046abd0580ef26"}
Nov 03 07:15:16.320636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.320330    4470 scope.go:122] "RemoveContainer" containerID="9ca75ff54d9e973ca7949a9425a7aca8453de15b037aeb794cef6f56a7709f8c"
Nov 03 07:15:16.320702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.320661    4470 scope.go:122] "RemoveContainer" containerID="47e9bc48b85f22364438b0ff1682295f8c3dd3f195fe6c4956046abd0580ef26"
Nov 03 07:15:16.320946 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:16.320924    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66_odh-ai-gateway-infra(003bacd2-9437-4842-b8c6-8abfc15ec54e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e"
Nov 03 07:15:16.321626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.321552    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"
Nov 03 07:15:16.321626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.321572    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f" event={"ID":"95072d50-f6f0-431a-b7e5-395a7c3dc2d7","Type":"ContainerDied","Data":"5eec3eebcb716d0a1b6657955eb50eb30982c48042464c1bd9bf8be48616fb95"}
Nov 03 07:15:16.321744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.321610    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eec3eebcb716d0a1b6657955eb50eb30982c48042464c1bd9bf8be48616fb95"
Nov 03 07:15:16.323472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.323447    4470 generic.go:356] "Generic (PLEG): container finished" podID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerID="c065829c1d3df962dce1cbb0020c894dc1fe1a2426769680d43fd3b37967819b" exitCode=60
Nov 03 07:15:16.323581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.323565    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h" event={"ID":"e57cbb60-e47b-4b77-a68c-5f09ba6ce906","Type":"ContainerDied","Data":"c065829c1d3df962dce1cbb0020c894dc1fe1a2426769680d43fd3b37967819b"}
Nov 03 07:15:16.323931 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.323914    4470 scope.go:122] "RemoveContainer" containerID="c065829c1d3df962dce1cbb0020c894dc1fe1a2426769680d43fd3b37967819b"
Nov 03 07:15:16.324209 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:16.324185    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h_odh-ai-gateway-infra(e57cbb60-e47b-4b77-a68c-5f09ba6ce906)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906"
Nov 03 07:15:16.333582 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.333556    4470 scope.go:122] "RemoveContainer" containerID="fc8e67941015bae58fb4450a08eb9cf925ec4a9c4ab0f52119f210bf95a1b36a"
Nov 03 07:15:16.478935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.478912    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"
Nov 03 07:15:16.561021 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.560989    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"]
Nov 03 07:15:16.584126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.584051    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cffac00c-a46c-4016-86b9-80ecfc91228e-kube-api-access-lxgct\" (UniqueName: \"kubernetes.io/projected/cffac00c-a46c-4016-86b9-80ecfc91228e-kube-api-access-lxgct\") pod \"cffac00c-a46c-4016-86b9-80ecfc91228e\" (UID: \"cffac00c-a46c-4016-86b9-80ecfc91228e\") "
Nov 03 07:15:16.584283 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.584186    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cffac00c-a46c-4016-86b9-80ecfc91228e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cffac00c-a46c-4016-86b9-80ecfc91228e-maas-api-service-ca\") pod \"cffac00c-a46c-4016-86b9-80ecfc91228e\" (UID: \"cffac00c-a46c-4016-86b9-80ecfc91228e\") "
Nov 03 07:15:16.584547 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.584516    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cffac00c-a46c-4016-86b9-80ecfc91228e-maas-api-service-ca" pod "cffac00c-a46c-4016-86b9-80ecfc91228e" (UID: "cffac00c-a46c-4016-86b9-80ecfc91228e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:15:16.586679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.586652    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffac00c-a46c-4016-86b9-80ecfc91228e-kube-api-access-lxgct" pod "cffac00c-a46c-4016-86b9-80ecfc91228e" (UID: "cffac00c-a46c-4016-86b9-80ecfc91228e"). InnerVolumeSpecName "kube-api-access-lxgct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:15:16.682796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.682762    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"]
Nov 03 07:15:16.686172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.686115    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxgct\" (UniqueName: \"kubernetes.io/projected/cffac00c-a46c-4016-86b9-80ecfc91228e-kube-api-access-lxgct\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:16.686172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.686172    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cffac00c-a46c-4016-86b9-80ecfc91228e-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:16.799471 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.799446    4470 scope.go:122] "RemoveContainer" containerID="8964266129f231dcf5ac76ede5f1cf45ac3dd8395304c5ef88ab813c98882621"
Nov 03 07:15:16.799654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:16.799638    4470 scope.go:122] "RemoveContainer" containerID="f2f6a26a43c4915ba94e766ec56a38128a8cef17e4e742b07b66ff5302c87669"
Nov 03 07:15:17.335301 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.335269    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"
Nov 03 07:15:17.335666 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.335278    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs" event={"ID":"cffac00c-a46c-4016-86b9-80ecfc91228e","Type":"ContainerDied","Data":"174f79e0e4f4f5844d09a2c63960a66bda352793fd84b1e785e033a616f7110a"}
Nov 03 07:15:17.335666 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.335443    4470 scope.go:122] "RemoveContainer" containerID="fb1cb62d18fcdf84a0723c49c05462bde9c5a349c0217b9ef5ff300850f6669c"
Nov 03 07:15:17.337591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.337567    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs" event={"ID":"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2","Type":"ContainerStarted","Data":"7d67e5ffc17100ce8a58d595c68dcfdfae76f77db635fac32c5fcee6053eca72"}
Nov 03 07:15:17.341595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.341568    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm" event={"ID":"28eb79bc-36bf-4ccc-ac22-e2d66597c572","Type":"ContainerStarted","Data":"715feb2e6b6e52812c48d7dab8d1a11df65ba04a2df6863ff8dc13b1f2245392"}
Nov 03 07:15:17.519653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.519625    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"
Nov 03 07:15:17.529815 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.529793    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"
Nov 03 07:15:17.601068 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.601043    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003bacd2-9437-4842-b8c6-8abfc15ec54e-kube-api-access-57m26\" (UniqueName: \"kubernetes.io/projected/003bacd2-9437-4842-b8c6-8abfc15ec54e-kube-api-access-57m26\") pod \"003bacd2-9437-4842-b8c6-8abfc15ec54e\" (UID: \"003bacd2-9437-4842-b8c6-8abfc15ec54e\") "
Nov 03 07:15:17.601234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.601089    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-kube-api-access-wqs6s\" (UniqueName: \"kubernetes.io/projected/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-kube-api-access-wqs6s\") pod \"e57cbb60-e47b-4b77-a68c-5f09ba6ce906\" (UID: \"e57cbb60-e47b-4b77-a68c-5f09ba6ce906\") "
Nov 03 07:15:17.601234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.601222    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-maas-api-service-ca\") pod \"e57cbb60-e47b-4b77-a68c-5f09ba6ce906\" (UID: \"e57cbb60-e47b-4b77-a68c-5f09ba6ce906\") "
Nov 03 07:15:17.601360 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.601254    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003bacd2-9437-4842-b8c6-8abfc15ec54e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003bacd2-9437-4842-b8c6-8abfc15ec54e-maas-api-service-ca\") pod \"003bacd2-9437-4842-b8c6-8abfc15ec54e\" (UID: \"003bacd2-9437-4842-b8c6-8abfc15ec54e\") "
Nov 03 07:15:17.601608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.601576    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-maas-api-service-ca" pod "e57cbb60-e47b-4b77-a68c-5f09ba6ce906" (UID: "e57cbb60-e47b-4b77-a68c-5f09ba6ce906"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:15:17.601608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.601599    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003bacd2-9437-4842-b8c6-8abfc15ec54e-maas-api-service-ca" pod "003bacd2-9437-4842-b8c6-8abfc15ec54e" (UID: "003bacd2-9437-4842-b8c6-8abfc15ec54e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:15:17.601783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.601746    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:17.601783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.601762    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003bacd2-9437-4842-b8c6-8abfc15ec54e-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:17.603571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.603542    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003bacd2-9437-4842-b8c6-8abfc15ec54e-kube-api-access-57m26" pod "003bacd2-9437-4842-b8c6-8abfc15ec54e" (UID: "003bacd2-9437-4842-b8c6-8abfc15ec54e"). InnerVolumeSpecName "kube-api-access-57m26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:15:17.603885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.603860    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-kube-api-access-wqs6s" pod "e57cbb60-e47b-4b77-a68c-5f09ba6ce906" (UID: "e57cbb60-e47b-4b77-a68c-5f09ba6ce906"). InnerVolumeSpecName "kube-api-access-wqs6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:15:17.703031 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.702955    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57m26\" (UniqueName: \"kubernetes.io/projected/003bacd2-9437-4842-b8c6-8abfc15ec54e-kube-api-access-57m26\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:17.703031 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.702981    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqs6s\" (UniqueName: \"kubernetes.io/projected/e57cbb60-e47b-4b77-a68c-5f09ba6ce906-kube-api-access-wqs6s\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:17.816494 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.816442    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs" podStartSLOduration=17.816428838 podStartE2EDuration="17.816428838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:17.815225872 +0000 UTC m=+970.147594866" watchObservedRunningTime="2026-11-03 07:15:17.816428838 +0000 UTC m=+970.148797869"
Nov 03 07:15:17.932445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.932416    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"]
Nov 03 07:15:17.940394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:17.940360    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894835-9vjfs"]
Nov 03 07:15:18.348704 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.348675    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"
Nov 03 07:15:18.349219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.348718    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h" event={"ID":"e57cbb60-e47b-4b77-a68c-5f09ba6ce906","Type":"ContainerDied","Data":"4a6c48a16274cc6d30845118502df6bb4bc1d09454a12e3f00df433dfd7623fb"}
Nov 03 07:15:18.349219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.348780    4470 scope.go:122] "RemoveContainer" containerID="c065829c1d3df962dce1cbb0020c894dc1fe1a2426769680d43fd3b37967819b"
Nov 03 07:15:18.352084 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.352064    4470 generic.go:356] "Generic (PLEG): container finished" podID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerID="7d67e5ffc17100ce8a58d595c68dcfdfae76f77db635fac32c5fcee6053eca72" exitCode=60
Nov 03 07:15:18.352206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.352144    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs" event={"ID":"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2","Type":"ContainerDied","Data":"7d67e5ffc17100ce8a58d595c68dcfdfae76f77db635fac32c5fcee6053eca72"}
Nov 03 07:15:18.352541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.352520    4470 scope.go:122] "RemoveContainer" containerID="7d67e5ffc17100ce8a58d595c68dcfdfae76f77db635fac32c5fcee6053eca72"
Nov 03 07:15:18.352818 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:18.352791    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs_odh-ai-gateway-infra(8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2"
Nov 03 07:15:18.353624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.353610    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"
Nov 03 07:15:18.353712 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.353632    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66" event={"ID":"003bacd2-9437-4842-b8c6-8abfc15ec54e","Type":"ContainerDied","Data":"c45f1a8e00af909e2cbf74b3147f2a992c4876feeadb18b0d289681617165606"}
Nov 03 07:15:18.356816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.356795    4470 generic.go:356] "Generic (PLEG): container finished" podID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerID="715feb2e6b6e52812c48d7dab8d1a11df65ba04a2df6863ff8dc13b1f2245392" exitCode=60
Nov 03 07:15:18.356907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.356847    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm" event={"ID":"28eb79bc-36bf-4ccc-ac22-e2d66597c572","Type":"ContainerDied","Data":"715feb2e6b6e52812c48d7dab8d1a11df65ba04a2df6863ff8dc13b1f2245392"}
Nov 03 07:15:18.357188 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.357173    4470 scope.go:122] "RemoveContainer" containerID="715feb2e6b6e52812c48d7dab8d1a11df65ba04a2df6863ff8dc13b1f2245392"
Nov 03 07:15:18.357433 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:18.357410    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm_odh-ai-gateway-infra(28eb79bc-36bf-4ccc-ac22-e2d66597c572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572"
Nov 03 07:15:18.359362 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.359346    4470 scope.go:122] "RemoveContainer" containerID="8964266129f231dcf5ac76ede5f1cf45ac3dd8395304c5ef88ab813c98882621"
Nov 03 07:15:18.371415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.371399    4470 scope.go:122] "RemoveContainer" containerID="47e9bc48b85f22364438b0ff1682295f8c3dd3f195fe6c4956046abd0580ef26"
Nov 03 07:15:18.379348 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.379332    4470 scope.go:122] "RemoveContainer" containerID="f2f6a26a43c4915ba94e766ec56a38128a8cef17e4e742b07b66ff5302c87669"
Nov 03 07:15:18.521666 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.521637    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"]
Nov 03 07:15:18.527927 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.527902    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894835-7z79h"]
Nov 03 07:15:18.588838 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.588809    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"]
Nov 03 07:15:18.760161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.760066    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"]
Nov 03 07:15:18.766766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.766742    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894835-ftl66"]
Nov 03 07:15:18.827215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:18.827186    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"]
Nov 03 07:15:19.520035 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.520009    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"
Nov 03 07:15:19.542778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.542757    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"
Nov 03 07:15:19.625205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.625093    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28eb79bc-36bf-4ccc-ac22-e2d66597c572-kube-api-access-j9fds\" (UniqueName: \"kubernetes.io/projected/28eb79bc-36bf-4ccc-ac22-e2d66597c572-kube-api-access-j9fds\") pod \"28eb79bc-36bf-4ccc-ac22-e2d66597c572\" (UID: \"28eb79bc-36bf-4ccc-ac22-e2d66597c572\") "
Nov 03 07:15:19.625205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.625188    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-kube-api-access-wbtqq\" (UniqueName: \"kubernetes.io/projected/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-kube-api-access-wbtqq\") pod \"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2\" (UID: \"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2\") "
Nov 03 07:15:19.625428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.625281    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-maas-api-service-ca\") pod \"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2\" (UID: \"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2\") "
Nov 03 07:15:19.625428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.625338    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28eb79bc-36bf-4ccc-ac22-e2d66597c572-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28eb79bc-36bf-4ccc-ac22-e2d66597c572-maas-api-service-ca\") pod \"28eb79bc-36bf-4ccc-ac22-e2d66597c572\" (UID: \"28eb79bc-36bf-4ccc-ac22-e2d66597c572\") "
Nov 03 07:15:19.625668 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.625639    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-maas-api-service-ca" pod "8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" (UID: "8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:15:19.625668 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.625664    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28eb79bc-36bf-4ccc-ac22-e2d66597c572-maas-api-service-ca" pod "28eb79bc-36bf-4ccc-ac22-e2d66597c572" (UID: "28eb79bc-36bf-4ccc-ac22-e2d66597c572"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:15:19.626050 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.625831    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28eb79bc-36bf-4ccc-ac22-e2d66597c572-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:19.626050 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.625845    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:19.627487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.627464    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28eb79bc-36bf-4ccc-ac22-e2d66597c572-kube-api-access-j9fds" pod "28eb79bc-36bf-4ccc-ac22-e2d66597c572" (UID: "28eb79bc-36bf-4ccc-ac22-e2d66597c572"). InnerVolumeSpecName "kube-api-access-j9fds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:15:19.627487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.627472    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-kube-api-access-wbtqq" pod "8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" (UID: "8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2"). InnerVolumeSpecName "kube-api-access-wbtqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:15:19.727372 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.727337    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9fds\" (UniqueName: \"kubernetes.io/projected/28eb79bc-36bf-4ccc-ac22-e2d66597c572-kube-api-access-j9fds\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:19.727372 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.727364    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbtqq\" (UniqueName: \"kubernetes.io/projected/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2-kube-api-access-wbtqq\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:19.806565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.806527    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e" path="/var/lib/kubelet/pods/003bacd2-9437-4842-b8c6-8abfc15ec54e/volumes"
Nov 03 07:15:19.806989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.806969    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e" path="/var/lib/kubelet/pods/cffac00c-a46c-4016-86b9-80ecfc91228e/volumes"
Nov 03 07:15:19.807393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:19.807376    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" path="/var/lib/kubelet/pods/e57cbb60-e47b-4b77-a68c-5f09ba6ce906/volumes"
Nov 03 07:15:20.374125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:20.374090    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"
Nov 03 07:15:20.374328 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:20.374121    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs" event={"ID":"8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2","Type":"ContainerDied","Data":"5cb4a1cee33f50b9c89eccb621624ddb705acdde0f69661c4bc801021b11eb12"}
Nov 03 07:15:20.374328 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:20.374217    4470 scope.go:122] "RemoveContainer" containerID="7d67e5ffc17100ce8a58d595c68dcfdfae76f77db635fac32c5fcee6053eca72"
Nov 03 07:15:20.375569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:20.375544    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm" event={"ID":"28eb79bc-36bf-4ccc-ac22-e2d66597c572","Type":"ContainerDied","Data":"4bd8b0b1ef3fae4bbaf8fd3dfc5c5d8d68821983804611e9b007eff8602704e4"}
Nov 03 07:15:20.375656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:20.375639    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"
Nov 03 07:15:20.383676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:20.383660    4470 scope.go:122] "RemoveContainer" containerID="715feb2e6b6e52812c48d7dab8d1a11df65ba04a2df6863ff8dc13b1f2245392"
Nov 03 07:15:20.546160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:20.546113    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"]
Nov 03 07:15:20.596879 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:20.596851    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894835-tgsfm"]
Nov 03 07:15:20.669548 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:20.669472    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"]
Nov 03 07:15:20.675416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:20.675394    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894835-4nczs"]
Nov 03 07:15:21.807102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:21.807058    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" path="/var/lib/kubelet/pods/28eb79bc-36bf-4ccc-ac22-e2d66597c572/volumes"
Nov 03 07:15:21.807541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:21.807519    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" path="/var/lib/kubelet/pods/8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2/volumes"
Nov 03 07:15:27.570774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.570741    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"]
Nov 03 07:15:27.571273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571260    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerName="cleanup"
Nov 03 07:15:27.571320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571274    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerName="cleanup"
Nov 03 07:15:27.571320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571291    4470 cpu_manager.go:414] "containerMap: removing container" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerName="cleanup"
Nov 03 07:15:27.571320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571296    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerName="cleanup"
Nov 03 07:15:27.571320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571304    4470 cpu_manager.go:414] "containerMap: removing container" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerName="cleanup"
Nov 03 07:15:27.571320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571309    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerName="cleanup"
Nov 03 07:15:27.571320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571314    4470 cpu_manager.go:414] "containerMap: removing container" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerName="cleanup"
Nov 03 07:15:27.571320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571319    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571330    4470 cpu_manager.go:414] "containerMap: removing container" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571335    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571341    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571345    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571351    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571355    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571364    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571368    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571379    4470 cpu_manager.go:414] "containerMap: removing container" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571383    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571391    4470 cpu_manager.go:414] "containerMap: removing container" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571395    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571407    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571411    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571419    4470 cpu_manager.go:414] "containerMap: removing container" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571422    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571436    4470 cpu_manager.go:414] "containerMap: removing container" podUID="95072d50-f6f0-431a-b7e5-395a7c3dc2d7" containerName="revoke-keys"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571439    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95072d50-f6f0-431a-b7e5-395a7c3dc2d7" containerName="revoke-keys"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571507    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571514    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571521    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571535    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerName="cleanup"
Nov 03 07:15:27.571525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571542    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerName="cleanup"
Nov 03 07:15:27.572503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571546    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerName="cleanup"
Nov 03 07:15:27.572503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571552    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerName="cleanup"
Nov 03 07:15:27.572503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571558    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerName="cleanup"
Nov 03 07:15:27.572503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571563    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerName="cleanup"
Nov 03 07:15:27.572503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571568    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerName="cleanup"
Nov 03 07:15:27.572503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571575    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerName="cleanup"
Nov 03 07:15:27.572503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571583    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="95072d50-f6f0-431a-b7e5-395a7c3dc2d7" containerName="revoke-keys"
Nov 03 07:15:27.572503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.571588    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerName="cleanup"
Nov 03 07:15:27.577104 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.577089    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:27.634340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.634315    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"]
Nov 03 07:15:27.681592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.681564    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:15:27.681592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.681572    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:15:27.681783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.681600    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:15:27.681783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.681639    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:15:27.709371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.709348    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1320b63a-fdgv8\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:27.709514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.709375    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1320b63a-fdgv8\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:27.709514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.709405    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5q2r\" (UniqueName: \"kubernetes.io/projected/e4971a2c-a168-41fb-8daa-ec350882a57f-kube-api-access-w5q2r\") pod \"maas-api-revoke-sub-1320b63a-fdgv8\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:27.810624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.810592    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1320b63a-fdgv8\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:27.810624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.810622    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1320b63a-fdgv8\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:27.810825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.810651    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5q2r\" (UniqueName: \"kubernetes.io/projected/e4971a2c-a168-41fb-8daa-ec350882a57f-kube-api-access-w5q2r\") pod \"maas-api-revoke-sub-1320b63a-fdgv8\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:27.811246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.811225    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1320b63a-fdgv8\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:27.813172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.813150    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1320b63a-fdgv8\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:27.874751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.874686    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5q2r\" (UniqueName: \"kubernetes.io/projected/e4971a2c-a168-41fb-8daa-ec350882a57f-kube-api-access-w5q2r\") pod \"maas-api-revoke-sub-1320b63a-fdgv8\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:27.890581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:27.890558    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:28.268519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:28.268492    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"]
Nov 03 07:15:28.270250 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:15:28.270221    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4971a2c_a168_41fb_8daa_ec350882a57f.slice/crio-c59df93a29415935b58ba4a50eb1b961181ab59e82a419089e5468dcf31320a4 WatchSource:0}: Error finding container c59df93a29415935b58ba4a50eb1b961181ab59e82a419089e5468dcf31320a4: Status 404 returned error can't find the container with id c59df93a29415935b58ba4a50eb1b961181ab59e82a419089e5468dcf31320a4
Nov 03 07:15:28.424709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:28.424675    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8" event={"ID":"e4971a2c-a168-41fb-8daa-ec350882a57f","Type":"ContainerStarted","Data":"c59df93a29415935b58ba4a50eb1b961181ab59e82a419089e5468dcf31320a4"}
Nov 03 07:15:29.431925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:29.431841    4470 generic.go:356] "Generic (PLEG): container finished" podID="e4971a2c-a168-41fb-8daa-ec350882a57f" containerID="78ab9509a5e9486060afec73406e0ed8c103f2b61ae83117191e1d1eb842c7b0" exitCode=0
Nov 03 07:15:29.432371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:29.431928    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8" event={"ID":"e4971a2c-a168-41fb-8daa-ec350882a57f","Type":"ContainerDied","Data":"78ab9509a5e9486060afec73406e0ed8c103f2b61ae83117191e1d1eb842c7b0"}
Nov 03 07:15:30.582824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:30.582800    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:30.639435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:30.639400    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-xks-ca\") pod \"e4971a2c-a168-41fb-8daa-ec350882a57f\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") "
Nov 03 07:15:30.639622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:30.639465    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4971a2c-a168-41fb-8daa-ec350882a57f-kube-api-access-w5q2r\" (UniqueName: \"kubernetes.io/projected/e4971a2c-a168-41fb-8daa-ec350882a57f-kube-api-access-w5q2r\") pod \"e4971a2c-a168-41fb-8daa-ec350882a57f\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") "
Nov 03 07:15:30.639622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:30.639545    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-openshift-ca\") pod \"e4971a2c-a168-41fb-8daa-ec350882a57f\" (UID: \"e4971a2c-a168-41fb-8daa-ec350882a57f\") "
Nov 03 07:15:30.639880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:30.639858    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-openshift-ca" pod "e4971a2c-a168-41fb-8daa-ec350882a57f" (UID: "e4971a2c-a168-41fb-8daa-ec350882a57f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:15:30.640059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:30.640038    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:30.641687 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:30.641666    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-xks-ca" pod "e4971a2c-a168-41fb-8daa-ec350882a57f" (UID: "e4971a2c-a168-41fb-8daa-ec350882a57f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:15:30.641806 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:30.641791    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4971a2c-a168-41fb-8daa-ec350882a57f-kube-api-access-w5q2r" pod "e4971a2c-a168-41fb-8daa-ec350882a57f" (UID: "e4971a2c-a168-41fb-8daa-ec350882a57f"). InnerVolumeSpecName "kube-api-access-w5q2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:15:30.741263 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:30.741184    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4971a2c-a168-41fb-8daa-ec350882a57f-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:30.741263 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:30.741220    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5q2r\" (UniqueName: \"kubernetes.io/projected/e4971a2c-a168-41fb-8daa-ec350882a57f-kube-api-access-w5q2r\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:31.446125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:31.446084    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8" event={"ID":"e4971a2c-a168-41fb-8daa-ec350882a57f","Type":"ContainerDied","Data":"c59df93a29415935b58ba4a50eb1b961181ab59e82a419089e5468dcf31320a4"}
Nov 03 07:15:31.446360 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:31.446181    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c59df93a29415935b58ba4a50eb1b961181ab59e82a419089e5468dcf31320a4"
Nov 03 07:15:31.446360 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:31.446254    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"
Nov 03 07:15:34.759288 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:15:34.759246    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:15:47.737955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.737922    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"]
Nov 03 07:15:47.738473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738459    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerName="cleanup"
Nov 03 07:15:47.738529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738474    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerName="cleanup"
Nov 03 07:15:47.738529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738499    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e4971a2c-a168-41fb-8daa-ec350882a57f" containerName="revoke-keys"
Nov 03 07:15:47.738529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738504    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4971a2c-a168-41fb-8daa-ec350882a57f" containerName="revoke-keys"
Nov 03 07:15:47.738529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738514    4470 cpu_manager.go:414] "containerMap: removing container" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerName="cleanup"
Nov 03 07:15:47.738529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738519    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003bacd2-9437-4842-b8c6-8abfc15ec54e" containerName="cleanup"
Nov 03 07:15:47.738670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738550    4470 cpu_manager.go:414] "containerMap: removing container" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerName="cleanup"
Nov 03 07:15:47.738670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738554    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28eb79bc-36bf-4ccc-ac22-e2d66597c572" containerName="cleanup"
Nov 03 07:15:47.738670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738637    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8442daaf-3bdd-4b9f-9d54-e8ad9a1eb0f2" containerName="cleanup"
Nov 03 07:15:47.738670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738646    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="cffac00c-a46c-4016-86b9-80ecfc91228e" containerName="cleanup"
Nov 03 07:15:47.738670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738658    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4971a2c-a168-41fb-8daa-ec350882a57f" containerName="revoke-keys"
Nov 03 07:15:47.738670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.738664    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57cbb60-e47b-4b77-a68c-5f09ba6ce906" containerName="cleanup"
Nov 03 07:15:47.742393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.742376    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:47.820299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.820259    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"]
Nov 03 07:15:47.823604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.823561    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f8c1943-wfprl\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:47.823778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.823757    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vbhh\" (UniqueName: \"kubernetes.io/projected/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-kube-api-access-4vbhh\") pod \"maas-api-revoke-sub-3f8c1943-wfprl\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:47.823861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.823809    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f8c1943-wfprl\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:47.848669 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.848619    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:15:47.848669 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.848633    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:15:47.848883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.848712    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:15:47.848883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.848786    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:15:47.925476 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.925442    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vbhh\" (UniqueName: \"kubernetes.io/projected/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-kube-api-access-4vbhh\") pod \"maas-api-revoke-sub-3f8c1943-wfprl\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:47.925659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.925502    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f8c1943-wfprl\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:47.925659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.925553    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f8c1943-wfprl\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:47.926231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.926202    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f8c1943-wfprl\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:47.928086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.928063    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f8c1943-wfprl\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:47.989962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:47.989887    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vbhh\" (UniqueName: \"kubernetes.io/projected/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-kube-api-access-4vbhh\") pod \"maas-api-revoke-sub-3f8c1943-wfprl\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:48.055744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:48.055710    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:48.441567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:48.441543    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"]
Nov 03 07:15:48.442944 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:15:48.442922    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae16c5f0_15ee_4899_83a0_41b3bb8de8d4.slice/crio-343f0e5316a8761ff43c7e853ac1f723ceb40a11422e6d927baea372a2ed57b1 WatchSource:0}: Error finding container 343f0e5316a8761ff43c7e853ac1f723ceb40a11422e6d927baea372a2ed57b1: Status 404 returned error can't find the container with id 343f0e5316a8761ff43c7e853ac1f723ceb40a11422e6d927baea372a2ed57b1
Nov 03 07:15:48.548716 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:48.548679    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl" event={"ID":"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4","Type":"ContainerStarted","Data":"343f0e5316a8761ff43c7e853ac1f723ceb40a11422e6d927baea372a2ed57b1"}
Nov 03 07:15:49.556297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:49.556263    4470 generic.go:356] "Generic (PLEG): container finished" podID="ae16c5f0-15ee-4899-83a0-41b3bb8de8d4" containerID="9fec5ac88e829020c4d4f91fe52d200420d273f2a7af17133ccfecd6cc5c4824" exitCode=0
Nov 03 07:15:49.556691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:49.556348    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl" event={"ID":"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4","Type":"ContainerDied","Data":"9fec5ac88e829020c4d4f91fe52d200420d273f2a7af17133ccfecd6cc5c4824"}
Nov 03 07:15:50.723166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:50.723119    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:50.860887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:50.860811    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-openshift-ca\") pod \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") "
Nov 03 07:15:50.861041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:50.860951    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-kube-api-access-4vbhh\" (UniqueName: \"kubernetes.io/projected/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-kube-api-access-4vbhh\") pod \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") "
Nov 03 07:15:50.861041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:50.860999    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-xks-ca\") pod \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\" (UID: \"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4\") "
Nov 03 07:15:50.861213 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:50.861168    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-openshift-ca" pod "ae16c5f0-15ee-4899-83a0-41b3bb8de8d4" (UID: "ae16c5f0-15ee-4899-83a0-41b3bb8de8d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:15:50.861432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:50.861414    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:50.863512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:50.863483    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-xks-ca" pod "ae16c5f0-15ee-4899-83a0-41b3bb8de8d4" (UID: "ae16c5f0-15ee-4899-83a0-41b3bb8de8d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:15:50.868584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:50.868556    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-kube-api-access-4vbhh" pod "ae16c5f0-15ee-4899-83a0-41b3bb8de8d4" (UID: "ae16c5f0-15ee-4899-83a0-41b3bb8de8d4"). InnerVolumeSpecName "kube-api-access-4vbhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:15:50.962918 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:50.962867    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:50.962918 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:50.962895    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vbhh\" (UniqueName: \"kubernetes.io/projected/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4-kube-api-access-4vbhh\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:51.570391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:51.570363    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"
Nov 03 07:15:51.570560 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:51.570395    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl" event={"ID":"ae16c5f0-15ee-4899-83a0-41b3bb8de8d4","Type":"ContainerDied","Data":"343f0e5316a8761ff43c7e853ac1f723ceb40a11422e6d927baea372a2ed57b1"}
Nov 03 07:15:51.570560 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:51.570443    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="343f0e5316a8761ff43c7e853ac1f723ceb40a11422e6d927baea372a2ed57b1"
Nov 03 07:15:53.603844 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.603807    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"]
Nov 03 07:15:53.604465 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.604449    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ae16c5f0-15ee-4899-83a0-41b3bb8de8d4" containerName="revoke-keys"
Nov 03 07:15:53.604465 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.604466    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae16c5f0-15ee-4899-83a0-41b3bb8de8d4" containerName="revoke-keys"
Nov 03 07:15:53.604591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.604577    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae16c5f0-15ee-4899-83a0-41b3bb8de8d4" containerName="revoke-keys"
Nov 03 07:15:53.607840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.607825    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:53.666598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.666563    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"]
Nov 03 07:15:53.690958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.690922    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05473694-698kj\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:53.691201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.690994    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gml88\" (UniqueName: \"kubernetes.io/projected/166eb2cf-6201-47ae-a18c-7ef14098bf4b-kube-api-access-gml88\") pod \"maas-api-revoke-sub-05473694-698kj\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:53.691201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.691105    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05473694-698kj\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:53.714381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.714357    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:15:53.714381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.714360    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:15:53.714381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.714359    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:15:53.714578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.714401    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:15:53.792113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.792080    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05473694-698kj\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:53.792308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.792160    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gml88\" (UniqueName: \"kubernetes.io/projected/166eb2cf-6201-47ae-a18c-7ef14098bf4b-kube-api-access-gml88\") pod \"maas-api-revoke-sub-05473694-698kj\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:53.792308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.792202    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05473694-698kj\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:53.792822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.792803    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05473694-698kj\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:53.794531 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.794510    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05473694-698kj\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:53.857807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.857723    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gml88\" (UniqueName: \"kubernetes.io/projected/166eb2cf-6201-47ae-a18c-7ef14098bf4b-kube-api-access-gml88\") pod \"maas-api-revoke-sub-05473694-698kj\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:53.921241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:53.921212    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:54.298092 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:54.298064    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"]
Nov 03 07:15:54.298877 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:15:54.298850    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166eb2cf_6201_47ae_a18c_7ef14098bf4b.slice/crio-42d2cbe4aed19e4e8b85241a3e0e3013bad4993c8d74622e5d2f8155a1559846 WatchSource:0}: Error finding container 42d2cbe4aed19e4e8b85241a3e0e3013bad4993c8d74622e5d2f8155a1559846: Status 404 returned error can't find the container with id 42d2cbe4aed19e4e8b85241a3e0e3013bad4993c8d74622e5d2f8155a1559846
Nov 03 07:15:54.590324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:54.590284    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj" event={"ID":"166eb2cf-6201-47ae-a18c-7ef14098bf4b","Type":"ContainerStarted","Data":"e4896ba790824625f0e62e6560f476985f5ec9801a79eb78a8b0420661706050"}
Nov 03 07:15:54.590457 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:54.590335    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj" event={"ID":"166eb2cf-6201-47ae-a18c-7ef14098bf4b","Type":"ContainerStarted","Data":"42d2cbe4aed19e4e8b85241a3e0e3013bad4993c8d74622e5d2f8155a1559846"}
Nov 03 07:15:54.713401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:54.713344    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj" podStartSLOduration=1.713328097 podStartE2EDuration="1.713328097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:54.710977562 +0000 UTC m=+1007.043346555" watchObservedRunningTime="2026-11-03 07:15:54.713328097 +0000 UTC m=+1007.045697090"
Nov 03 07:15:55.598049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:55.598008    4470 generic.go:356] "Generic (PLEG): container finished" podID="166eb2cf-6201-47ae-a18c-7ef14098bf4b" containerID="e4896ba790824625f0e62e6560f476985f5ec9801a79eb78a8b0420661706050" exitCode=0
Nov 03 07:15:55.598219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:55.598045    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj" event={"ID":"166eb2cf-6201-47ae-a18c-7ef14098bf4b","Type":"ContainerDied","Data":"e4896ba790824625f0e62e6560f476985f5ec9801a79eb78a8b0420661706050"}
Nov 03 07:15:56.752825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:56.752799    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:56.926085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:56.926007    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-openshift-ca\") pod \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") "
Nov 03 07:15:56.926262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:56.926153    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166eb2cf-6201-47ae-a18c-7ef14098bf4b-kube-api-access-gml88\" (UniqueName: \"kubernetes.io/projected/166eb2cf-6201-47ae-a18c-7ef14098bf4b-kube-api-access-gml88\") pod \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") "
Nov 03 07:15:56.926262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:56.926204    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-xks-ca\") pod \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\" (UID: \"166eb2cf-6201-47ae-a18c-7ef14098bf4b\") "
Nov 03 07:15:56.926387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:56.926362    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-openshift-ca" pod "166eb2cf-6201-47ae-a18c-7ef14098bf4b" (UID: "166eb2cf-6201-47ae-a18c-7ef14098bf4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:15:56.926573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:56.926554    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:56.928325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:56.928295    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-xks-ca" pod "166eb2cf-6201-47ae-a18c-7ef14098bf4b" (UID: "166eb2cf-6201-47ae-a18c-7ef14098bf4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:15:56.928431 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:56.928359    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166eb2cf-6201-47ae-a18c-7ef14098bf4b-kube-api-access-gml88" pod "166eb2cf-6201-47ae-a18c-7ef14098bf4b" (UID: "166eb2cf-6201-47ae-a18c-7ef14098bf4b"). InnerVolumeSpecName "kube-api-access-gml88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:15:57.027817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:57.027783    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gml88\" (UniqueName: \"kubernetes.io/projected/166eb2cf-6201-47ae-a18c-7ef14098bf4b-kube-api-access-gml88\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:57.027817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:57.027809    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166eb2cf-6201-47ae-a18c-7ef14098bf4b-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:15:57.612024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:57.612000    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"
Nov 03 07:15:57.612213 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:57.611997    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj" event={"ID":"166eb2cf-6201-47ae-a18c-7ef14098bf4b","Type":"ContainerDied","Data":"42d2cbe4aed19e4e8b85241a3e0e3013bad4993c8d74622e5d2f8155a1559846"}
Nov 03 07:15:57.612213 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:57.612128    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42d2cbe4aed19e4e8b85241a3e0e3013bad4993c8d74622e5d2f8155a1559846"
Nov 03 07:15:58.551161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.551115    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"]
Nov 03 07:15:58.551640 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.551625    4470 cpu_manager.go:414] "containerMap: removing container" podUID="166eb2cf-6201-47ae-a18c-7ef14098bf4b" containerName="revoke-keys"
Nov 03 07:15:58.551640 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.551636    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166eb2cf-6201-47ae-a18c-7ef14098bf4b" containerName="revoke-keys"
Nov 03 07:15:58.551757 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.551747    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="166eb2cf-6201-47ae-a18c-7ef14098bf4b" containerName="revoke-keys"
Nov 03 07:15:58.554695 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.554680    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:15:58.613966 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.613937    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"]
Nov 03 07:15:58.661039 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.661007    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:15:58.661285 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.661159    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:15:58.661285 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.661187    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:15:58.661285 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.661203    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:15:58.745387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.745353    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaedccdc-mhzhm\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:15:58.745573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.745492    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rk2\" (UniqueName: \"kubernetes.io/projected/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-kube-api-access-98rk2\") pod \"maas-api-revoke-sub-aaedccdc-mhzhm\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:15:58.745673 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.745652    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaedccdc-mhzhm\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:15:58.848500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.848465    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaedccdc-mhzhm\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:15:58.848700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.848536    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaedccdc-mhzhm\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:15:58.848700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.848622    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rk2\" (UniqueName: \"kubernetes.io/projected/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-kube-api-access-98rk2\") pod \"maas-api-revoke-sub-aaedccdc-mhzhm\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:15:58.849452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.849395    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaedccdc-mhzhm\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:15:58.851107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.851082    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaedccdc-mhzhm\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:15:58.912723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:58.912695    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rk2\" (UniqueName: \"kubernetes.io/projected/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-kube-api-access-98rk2\") pod \"maas-api-revoke-sub-aaedccdc-mhzhm\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:15:59.168458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:59.168367    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:15:59.544706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:59.544674    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"]
Nov 03 07:15:59.545950 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:15:59.545923    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e1c2a0_e24c_46d3_a0a9_bd4af7ea33f1.slice/crio-a593b4aa60a907fe36d500cea57e273ef30358ba88d7c0a0692505b3612a6d13 WatchSource:0}: Error finding container a593b4aa60a907fe36d500cea57e273ef30358ba88d7c0a0692505b3612a6d13: Status 404 returned error can't find the container with id a593b4aa60a907fe36d500cea57e273ef30358ba88d7c0a0692505b3612a6d13
Nov 03 07:15:59.626539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:15:59.626502    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm" event={"ID":"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1","Type":"ContainerStarted","Data":"a593b4aa60a907fe36d500cea57e273ef30358ba88d7c0a0692505b3612a6d13"}
Nov 03 07:16:00.633987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:00.633955    4470 generic.go:356] "Generic (PLEG): container finished" podID="f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1" containerID="86cbb4a9449692ed6333c7ca2c87172c241590d420d64e0666171d743cb9f727" exitCode=0
Nov 03 07:16:00.634429 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:00.634039    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm" event={"ID":"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1","Type":"ContainerDied","Data":"86cbb4a9449692ed6333c7ca2c87172c241590d420d64e0666171d743cb9f727"}
Nov 03 07:16:01.508662 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.508628    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-9hf2w-pxwm6"]
Nov 03 07:16:01.512280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.512265    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.572149 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.572106    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9hf2w-pxwm6"]
Nov 03 07:16:01.619789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.619747    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 07:16:01.619789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.619792    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8gp29\""
Nov 03 07:16:01.681399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.681322    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-trusted-ca-bundle\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.681740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.681442    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blqr7\" (UniqueName: \"kubernetes.io/projected/913409c5-9471-495c-8556-92596081537d-kube-api-access-blqr7\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.681740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.681530    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-service-ca-bundle\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.681740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.681605    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/913409c5-9471-495c-8556-92596081537d-archives-path\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.762931 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.762899    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 07:16:01.782446 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.782416    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-service-ca-bundle\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.782591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.782465    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/913409c5-9471-495c-8556-92596081537d-archives-path\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.782591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.782532    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-trusted-ca-bundle\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.782591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.782571    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blqr7\" (UniqueName: \"kubernetes.io/projected/913409c5-9471-495c-8556-92596081537d-kube-api-access-blqr7\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.783005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.782981    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/913409c5-9471-495c-8556-92596081537d-archives-path\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.783210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.783191    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-service-ca-bundle\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.783399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.783384    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-trusted-ca-bundle\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.787659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.787643    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:16:01.846675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.846645    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blqr7\" (UniqueName: \"kubernetes.io/projected/913409c5-9471-495c-8556-92596081537d-kube-api-access-blqr7\") pod \"periodic-gathering-9hf2w-pxwm6\" (UID: \"913409c5-9471-495c-8556-92596081537d\") " pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:01.984653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.984573    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-xks-ca\") pod \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") "
Nov 03 07:16:01.984823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.984697    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-openshift-ca\") pod \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") "
Nov 03 07:16:01.984823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.984816    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-kube-api-access-98rk2\" (UniqueName: \"kubernetes.io/projected/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-kube-api-access-98rk2\") pod \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\" (UID: \"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1\") "
Nov 03 07:16:01.985106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.985078    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-openshift-ca" pod "f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1" (UID: "f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:01.985298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.985283    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:01.986952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.986930    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-xks-ca" pod "f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1" (UID: "f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:01.987034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:01.987016    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-kube-api-access-98rk2" pod "f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1" (UID: "f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1"). InnerVolumeSpecName "kube-api-access-98rk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:02.085890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:02.085855    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98rk2\" (UniqueName: \"kubernetes.io/projected/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-kube-api-access-98rk2\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:02.085890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:02.085885    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:02.124679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:02.124651    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:16:02.502291 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:02.502264    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9hf2w-pxwm6"]
Nov 03 07:16:02.503220 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:02.503192    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913409c5_9471_495c_8556_92596081537d.slice/crio-2272645670ac239a08e2e92ea2eaf0c66f4082fc3cfd177bc8931b95530b7202 WatchSource:0}: Error finding container 2272645670ac239a08e2e92ea2eaf0c66f4082fc3cfd177bc8931b95530b7202: Status 404 returned error can't find the container with id 2272645670ac239a08e2e92ea2eaf0c66f4082fc3cfd177bc8931b95530b7202
Nov 03 07:16:02.648240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:02.648205    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9hf2w-pxwm6" event={"ID":"913409c5-9471-495c-8556-92596081537d","Type":"ContainerStarted","Data":"2f41ab02aabc5c2a05f0e878258e687082389f6d1481151fe01ca20c968f278a"}
Nov 03 07:16:02.648410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:02.648278    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9hf2w-pxwm6" event={"ID":"913409c5-9471-495c-8556-92596081537d","Type":"ContainerStarted","Data":"2272645670ac239a08e2e92ea2eaf0c66f4082fc3cfd177bc8931b95530b7202"}
Nov 03 07:16:02.649686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:02.649660    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"
Nov 03 07:16:02.649802 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:02.649664    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm" event={"ID":"f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1","Type":"ContainerDied","Data":"a593b4aa60a907fe36d500cea57e273ef30358ba88d7c0a0692505b3612a6d13"}
Nov 03 07:16:02.649802 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:02.649774    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a593b4aa60a907fe36d500cea57e273ef30358ba88d7c0a0692505b3612a6d13"
Nov 03 07:16:02.769693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:02.769599    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-9hf2w-pxwm6" podStartSLOduration=1.7695859870000001 podStartE2EDuration="1.769585987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:02.767568217 +0000 UTC m=+1015.099937211" watchObservedRunningTime="2026-11-03 07:16:02.769585987 +0000 UTC m=+1015.101954982"
Nov 03 07:16:04.760108 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:16:04.760067    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:16:06.894524 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.894490    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"]
Nov 03 07:16:06.895007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.894994    4470 cpu_manager.go:414] "containerMap: removing container" podUID="f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1" containerName="revoke-keys"
Nov 03 07:16:06.895083 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.895008    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1" containerName="revoke-keys"
Nov 03 07:16:06.895178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.895096    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1" containerName="revoke-keys"
Nov 03 07:16:06.898599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.898581    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:06.934979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.934943    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:06.935107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.934994    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:06.935107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.935078    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:06.935219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.935112    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:06.935255 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.935221    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:06.935322 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.935306    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m997f\" (UniqueName: \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-kube-api-access-m997f\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:06.935372 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.935356    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:06.935428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.935415    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:06.935463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.935440    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:06.954937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:06.954916    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-669d6b-openshift-default-dockercfg-f5qrn\""
Nov 03 07:16:07.009472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.009443    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"]
Nov 03 07:16:07.036101 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.036069    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.036310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.036121    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.036310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.036203    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.036310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.036249    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m997f\" (UniqueName: \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-kube-api-access-m997f\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.036310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.036288    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.036526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.036357    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.036526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.036383    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.036526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.036428    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.036526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.036484    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.037779 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.037039    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.037779 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.037298    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.037779 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.037532    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.038004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.037891    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.038397 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.038367    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.039094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.039066    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.047161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.040754    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.101414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.101377    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.101560 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.101547    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m997f\" (UniqueName: \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-kube-api-access-m997f\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.212693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.212606    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:07.659539 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:07.659493    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dbf6a0_8d6e_4cbc_bde3_7b665e9021df.slice/crio-2df2f34b64f7e83f6531e80be5071236a963c8cc2cda4e2ac2511149a1af5a9a WatchSource:0}: Error finding container 2df2f34b64f7e83f6531e80be5071236a963c8cc2cda4e2ac2511149a1af5a9a: Status 404 returned error can't find the container with id 2df2f34b64f7e83f6531e80be5071236a963c8cc2cda4e2ac2511149a1af5a9a
Nov 03 07:16:07.659665 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.659613    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"]
Nov 03 07:16:07.661842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.661812    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:16:07.661917 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.661879    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:16:07.661917 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.661907    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:16:07.681899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:07.681874    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr" event={"ID":"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df","Type":"ContainerStarted","Data":"2df2f34b64f7e83f6531e80be5071236a963c8cc2cda4e2ac2511149a1af5a9a"}
Nov 03 07:16:08.688691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:08.688653    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr" event={"ID":"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df","Type":"ContainerStarted","Data":"4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81"}
Nov 03 07:16:09.213696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:09.213664    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:09.215415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:09.215390    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 07:16:09.215549 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:09.215434    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr" podUID="59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 07:16:10.213530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:10.213497    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 07:16:10.213932 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:10.213553    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr" podUID="59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 07:16:11.213520 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:11.213489    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 07:16:11.213695 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:11.213538    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr" podUID="59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 07:16:12.213474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:12.213429    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 07:16:12.213668 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:12.213491    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr" podUID="59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 07:16:13.213844 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:13.213809    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 07:16:13.214436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:13.213862    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr" podUID="59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 07:16:14.289486 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:14.289449    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:14.290218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:14.290195    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:14.291043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:14.291018    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:16:14.467053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:14.467003    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr" podStartSLOduration=8.466988316 podStartE2EDuration="8.466988316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:08.868465191 +0000 UTC m=+1021.200834212" watchObservedRunningTime="2026-11-03 07:16:14.466988316 +0000 UTC m=+1026.799357307"
Nov 03 07:16:19.359380 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.359307    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"]
Nov 03 07:16:19.364766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.364750    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:19.423169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.423125    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"]
Nov 03 07:16:19.469834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.469802    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:16:19.469834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.469822    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:16:19.469834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.469836    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:16:19.470091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.469935    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:16:19.501438 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.501396    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-130a2d11-tgj7h\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:19.501438 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.501436    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqkn\" (UniqueName: \"kubernetes.io/projected/a9494ded-12c4-4754-91c8-6662ada16b8d-kube-api-access-2xqkn\") pod \"maas-api-revoke-sub-130a2d11-tgj7h\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:19.501647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.501572    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-130a2d11-tgj7h\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:19.602967 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.602930    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-130a2d11-tgj7h\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:19.603164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.603069    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-130a2d11-tgj7h\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:19.603164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.603092    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqkn\" (UniqueName: \"kubernetes.io/projected/a9494ded-12c4-4754-91c8-6662ada16b8d-kube-api-access-2xqkn\") pod \"maas-api-revoke-sub-130a2d11-tgj7h\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:19.603596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.603576    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-130a2d11-tgj7h\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:19.605608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.605588    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-130a2d11-tgj7h\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:19.667725 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.667648    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqkn\" (UniqueName: \"kubernetes.io/projected/a9494ded-12c4-4754-91c8-6662ada16b8d-kube-api-access-2xqkn\") pod \"maas-api-revoke-sub-130a2d11-tgj7h\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:19.678525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:19.678501    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:20.055316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:20.055287    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"]
Nov 03 07:16:20.056552 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:20.056528    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9494ded_12c4_4754_91c8_6662ada16b8d.slice/crio-c975ae9fc2e58ac35dc9f5572ea62237e976f7bf8cb238611c1dad1f0b461639 WatchSource:0}: Error finding container c975ae9fc2e58ac35dc9f5572ea62237e976f7bf8cb238611c1dad1f0b461639: Status 404 returned error can't find the container with id c975ae9fc2e58ac35dc9f5572ea62237e976f7bf8cb238611c1dad1f0b461639
Nov 03 07:16:20.765459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:20.765360    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h" event={"ID":"a9494ded-12c4-4754-91c8-6662ada16b8d","Type":"ContainerStarted","Data":"b62a6030d64f786bfb6a7a876255f69dd0c47a2b4159ce1a71424c5eb3df92f9"}
Nov 03 07:16:20.765459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:20.765410    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h" event={"ID":"a9494ded-12c4-4754-91c8-6662ada16b8d","Type":"ContainerStarted","Data":"c975ae9fc2e58ac35dc9f5572ea62237e976f7bf8cb238611c1dad1f0b461639"}
Nov 03 07:16:20.892413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:20.892352    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h" podStartSLOduration=1.892337504 podStartE2EDuration="1.892337504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:20.891076639 +0000 UTC m=+1033.223445634" watchObservedRunningTime="2026-11-03 07:16:20.892337504 +0000 UTC m=+1033.224706498"
Nov 03 07:16:21.771850 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:21.771818    4470 generic.go:356] "Generic (PLEG): container finished" podID="a9494ded-12c4-4754-91c8-6662ada16b8d" containerID="b62a6030d64f786bfb6a7a876255f69dd0c47a2b4159ce1a71424c5eb3df92f9" exitCode=0
Nov 03 07:16:21.772296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:21.771909    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h" event={"ID":"a9494ded-12c4-4754-91c8-6662ada16b8d","Type":"ContainerDied","Data":"b62a6030d64f786bfb6a7a876255f69dd0c47a2b4159ce1a71424c5eb3df92f9"}
Nov 03 07:16:22.024464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.024378    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"]
Nov 03 07:16:22.028284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.028267    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.083790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.083761    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6606d6-openshift-default-dockercfg-hxk2h\""
Nov 03 07:16:22.091825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.091791    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"]
Nov 03 07:16:22.153502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.153462    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.153661 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.153571    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.153661 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.153601    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.153661 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.153619    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.153799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.153703    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflmk\" (UniqueName: \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-kube-api-access-rflmk\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.153799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.153724    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.153876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.153822    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.153956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.153941    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.153996 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.153982    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.254624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.254584    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.254792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.254641    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.254792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.254673    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.254880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.254836    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.254880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.254869    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.254991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.254886    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.254991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.254909    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflmk\" (UniqueName: \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-kube-api-access-rflmk\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.254991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.254934    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.254991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.254979    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.255247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.255098    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.255247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.255121    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.255365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.255241    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.255422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.255376    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.255666 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.255642    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.257278 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.257259    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.257421 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.257405    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.319695 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.319661    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.319842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.319765    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflmk\" (UniqueName: \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-kube-api-access-rflmk\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.342291 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.342264    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:22.771183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.771158    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"]
Nov 03 07:16:22.772722 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:22.772688    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7011aff5_3fda_4fd7_ac52_ec1dc99205e8.slice/crio-7bec4985f9a97bc786e772be8b90a59ea9470fd6da8179e750ec314c6b1355a1 WatchSource:0}: Error finding container 7bec4985f9a97bc786e772be8b90a59ea9470fd6da8179e750ec314c6b1355a1: Status 404 returned error can't find the container with id 7bec4985f9a97bc786e772be8b90a59ea9470fd6da8179e750ec314c6b1355a1
Nov 03 07:16:22.775051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.775018    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:16:22.775125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.775090    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:16:22.775190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.775124    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:16:22.915795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:22.915767    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:23.063828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.063794    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-xks-ca\") pod \"a9494ded-12c4-4754-91c8-6662ada16b8d\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") "
Nov 03 07:16:23.063989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.063887    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9494ded-12c4-4754-91c8-6662ada16b8d-kube-api-access-2xqkn\" (UniqueName: \"kubernetes.io/projected/a9494ded-12c4-4754-91c8-6662ada16b8d-kube-api-access-2xqkn\") pod \"a9494ded-12c4-4754-91c8-6662ada16b8d\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") "
Nov 03 07:16:23.064085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.064062    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-openshift-ca\") pod \"a9494ded-12c4-4754-91c8-6662ada16b8d\" (UID: \"a9494ded-12c4-4754-91c8-6662ada16b8d\") "
Nov 03 07:16:23.064517 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.064486    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-openshift-ca" pod "a9494ded-12c4-4754-91c8-6662ada16b8d" (UID: "a9494ded-12c4-4754-91c8-6662ada16b8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:23.064649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.064606    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:23.066377 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.066352    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9494ded-12c4-4754-91c8-6662ada16b8d-kube-api-access-2xqkn" pod "a9494ded-12c4-4754-91c8-6662ada16b8d" (UID: "a9494ded-12c4-4754-91c8-6662ada16b8d"). InnerVolumeSpecName "kube-api-access-2xqkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:23.066479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.066462    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-xks-ca" pod "a9494ded-12c4-4754-91c8-6662ada16b8d" (UID: "a9494ded-12c4-4754-91c8-6662ada16b8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:23.166314 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.166285    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xqkn\" (UniqueName: \"kubernetes.io/projected/a9494ded-12c4-4754-91c8-6662ada16b8d-kube-api-access-2xqkn\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:23.166314 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.166310    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9494ded-12c4-4754-91c8-6662ada16b8d-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:23.788689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.788655    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" event={"ID":"7011aff5-3fda-4fd7-ac52-ec1dc99205e8","Type":"ContainerStarted","Data":"22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1"}
Nov 03 07:16:23.789183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.788715    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" event={"ID":"7011aff5-3fda-4fd7-ac52-ec1dc99205e8","Type":"ContainerStarted","Data":"7bec4985f9a97bc786e772be8b90a59ea9470fd6da8179e750ec314c6b1355a1"}
Nov 03 07:16:23.790210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.790188    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h" event={"ID":"a9494ded-12c4-4754-91c8-6662ada16b8d","Type":"ContainerDied","Data":"c975ae9fc2e58ac35dc9f5572ea62237e976f7bf8cb238611c1dad1f0b461639"}
Nov 03 07:16:23.790302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.790216    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"
Nov 03 07:16:23.790302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.790223    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c975ae9fc2e58ac35dc9f5572ea62237e976f7bf8cb238611c1dad1f0b461639"
Nov 03 07:16:23.985396 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:23.985345    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" podStartSLOduration=2.985332368 podStartE2EDuration="2.985332368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:23.971558805 +0000 UTC m=+1036.303927796" watchObservedRunningTime="2026-11-03 07:16:23.985332368 +0000 UTC m=+1036.317701361"
Nov 03 07:16:24.342454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:24.342419    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:24.343895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:24.343869    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 07:16:24.344030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:24.343916    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" podUID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 07:16:25.343738 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:25.343695    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 07:16:25.344212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:25.343758    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" podUID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 07:16:26.342744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:26.342702    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 07:16:26.342930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:26.342755    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" podUID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 07:16:27.343316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:27.343283    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 07:16:27.343755 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:27.343337    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" podUID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 07:16:28.342703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:28.342669    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 07:16:28.342899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:28.342721    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" podUID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 07:16:29.430060 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.430018    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"]
Nov 03 07:16:29.430996 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.430967    4470 cpu_manager.go:414] "containerMap: removing container" podUID="a9494ded-12c4-4754-91c8-6662ada16b8d" containerName="revoke-keys"
Nov 03 07:16:29.430996 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.430993    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9494ded-12c4-4754-91c8-6662ada16b8d" containerName="revoke-keys"
Nov 03 07:16:29.431279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.431193    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9494ded-12c4-4754-91c8-6662ada16b8d" containerName="revoke-keys"
Nov 03 07:16:29.439851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.439824    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:29.491212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.491169    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"]
Nov 03 07:16:29.536982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.536947    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:16:29.536982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.536978    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:16:29.537267 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.537002    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:16:29.543836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.543806    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4878ee-rkh22\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:29.544024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.544004    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4878ee-rkh22\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:29.544168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.544123    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mklh\" (UniqueName: \"kubernetes.io/projected/367d6236-a725-4e4d-a86d-68980e9c2672-kube-api-access-6mklh\") pod \"maas-api-revoke-sub-9e4878ee-rkh22\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:29.545677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.545662    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:16:29.599071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.599041    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:29.599566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.599544    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:29.600467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.600448    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:16:29.645841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.645811    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4878ee-rkh22\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:29.646000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.645946    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4878ee-rkh22\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:29.646060 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.646045    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mklh\" (UniqueName: \"kubernetes.io/projected/367d6236-a725-4e4d-a86d-68980e9c2672-kube-api-access-6mklh\") pod \"maas-api-revoke-sub-9e4878ee-rkh22\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:29.646760 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.646736    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4878ee-rkh22\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:29.648542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.648519    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4878ee-rkh22\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:29.710289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.710211    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mklh\" (UniqueName: \"kubernetes.io/projected/367d6236-a725-4e4d-a86d-68980e9c2672-kube-api-access-6mklh\") pod \"maas-api-revoke-sub-9e4878ee-rkh22\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:29.756639 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:29.756606    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:30.142075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:30.142051    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"]
Nov 03 07:16:30.144192 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:30.144165    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367d6236_a725_4e4d_a86d_68980e9c2672.slice/crio-38b898674a98c6d7c7c96ef5d678d8876e18bb7987a957425ab2da4fbaae5629 WatchSource:0}: Error finding container 38b898674a98c6d7c7c96ef5d678d8876e18bb7987a957425ab2da4fbaae5629: Status 404 returned error can't find the container with id 38b898674a98c6d7c7c96ef5d678d8876e18bb7987a957425ab2da4fbaae5629
Nov 03 07:16:30.851875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:30.851830    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22" event={"ID":"367d6236-a725-4e4d-a86d-68980e9c2672","Type":"ContainerStarted","Data":"7d03e13dd368218146b226e7d0fbdd841f8d9d7ba87bab06fe40af902d8105bb"}
Nov 03 07:16:30.852364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:30.851883    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22" event={"ID":"367d6236-a725-4e4d-a86d-68980e9c2672","Type":"ContainerStarted","Data":"38b898674a98c6d7c7c96ef5d678d8876e18bb7987a957425ab2da4fbaae5629"}
Nov 03 07:16:30.977195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:30.977104    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22" podStartSLOduration=1.9770839100000002 podStartE2EDuration="1.97708391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:30.974549828 +0000 UTC m=+1043.306918823" watchObservedRunningTime="2026-11-03 07:16:30.97708391 +0000 UTC m=+1043.309452906"
Nov 03 07:16:31.377410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.377377    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"]
Nov 03 07:16:31.381711 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.381690    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:31.441276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.441204    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"]
Nov 03 07:16:31.477701 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.477666    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808bedea-8kscb\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:31.477905 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.477783    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808bedea-8kscb\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:31.477980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.477929    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4bh\" (UniqueName: \"kubernetes.io/projected/4f7d54c2-c364-4b6c-8eb6-283b277046d1-kube-api-access-tz4bh\") pod \"maas-api-revoke-sub-808bedea-8kscb\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:31.581257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.581007    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4bh\" (UniqueName: \"kubernetes.io/projected/4f7d54c2-c364-4b6c-8eb6-283b277046d1-kube-api-access-tz4bh\") pod \"maas-api-revoke-sub-808bedea-8kscb\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:31.581257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.581118    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808bedea-8kscb\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:31.581257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.581226    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808bedea-8kscb\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:31.588682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.588656    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808bedea-8kscb\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:31.590276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.590250    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808bedea-8kscb\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:31.649269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.649241    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4bh\" (UniqueName: \"kubernetes.io/projected/4f7d54c2-c364-4b6c-8eb6-283b277046d1-kube-api-access-tz4bh\") pod \"maas-api-revoke-sub-808bedea-8kscb\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:31.706303 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.706228    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:31.861018 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.859775    4470 generic.go:356] "Generic (PLEG): container finished" podID="367d6236-a725-4e4d-a86d-68980e9c2672" containerID="7d03e13dd368218146b226e7d0fbdd841f8d9d7ba87bab06fe40af902d8105bb" exitCode=0
Nov 03 07:16:31.861018 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:31.859840    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22" event={"ID":"367d6236-a725-4e4d-a86d-68980e9c2672","Type":"ContainerDied","Data":"7d03e13dd368218146b226e7d0fbdd841f8d9d7ba87bab06fe40af902d8105bb"}
Nov 03 07:16:32.125154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:32.120432    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"]
Nov 03 07:16:32.868811 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:32.868777    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb" event={"ID":"4f7d54c2-c364-4b6c-8eb6-283b277046d1","Type":"ContainerStarted","Data":"bb8a8cd5177db1f978c5107ffab525febb76dde1b8456020773a6655831d9949"}
Nov 03 07:16:32.869763 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:32.869726    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb" event={"ID":"4f7d54c2-c364-4b6c-8eb6-283b277046d1","Type":"ContainerStarted","Data":"838da82e5d572d5a4b681cc4dfce0513606a47ecf8dfd60936d3487fe658a049"}
Nov 03 07:16:33.005165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.004833    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb" podStartSLOduration=2.004812925 podStartE2EDuration="2.004812925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:32.995069094 +0000 UTC m=+1045.327438089" watchObservedRunningTime="2026-11-03 07:16:33.004812925 +0000 UTC m=+1045.337181922"
Nov 03 07:16:33.076272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.071704    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:33.105502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.105470    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-xks-ca\") pod \"367d6236-a725-4e4d-a86d-68980e9c2672\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") "
Nov 03 07:16:33.105680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.105542    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367d6236-a725-4e4d-a86d-68980e9c2672-kube-api-access-6mklh\" (UniqueName: \"kubernetes.io/projected/367d6236-a725-4e4d-a86d-68980e9c2672-kube-api-access-6mklh\") pod \"367d6236-a725-4e4d-a86d-68980e9c2672\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") "
Nov 03 07:16:33.105744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.105713    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-openshift-ca\") pod \"367d6236-a725-4e4d-a86d-68980e9c2672\" (UID: \"367d6236-a725-4e4d-a86d-68980e9c2672\") "
Nov 03 07:16:33.106103 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.106073    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-openshift-ca" pod "367d6236-a725-4e4d-a86d-68980e9c2672" (UID: "367d6236-a725-4e4d-a86d-68980e9c2672"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:33.107866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.107832    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-xks-ca" pod "367d6236-a725-4e4d-a86d-68980e9c2672" (UID: "367d6236-a725-4e4d-a86d-68980e9c2672"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:33.107984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.107960    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367d6236-a725-4e4d-a86d-68980e9c2672-kube-api-access-6mklh" pod "367d6236-a725-4e4d-a86d-68980e9c2672" (UID: "367d6236-a725-4e4d-a86d-68980e9c2672"). InnerVolumeSpecName "kube-api-access-6mklh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:33.207773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.207738    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:33.207773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.207776    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mklh\" (UniqueName: \"kubernetes.io/projected/367d6236-a725-4e4d-a86d-68980e9c2672-kube-api-access-6mklh\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:33.207962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.207795    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367d6236-a725-4e4d-a86d-68980e9c2672-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:33.876245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.876198    4470 generic.go:356] "Generic (PLEG): container finished" podID="4f7d54c2-c364-4b6c-8eb6-283b277046d1" containerID="bb8a8cd5177db1f978c5107ffab525febb76dde1b8456020773a6655831d9949" exitCode=0
Nov 03 07:16:33.876700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.876290    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb" event={"ID":"4f7d54c2-c364-4b6c-8eb6-283b277046d1","Type":"ContainerDied","Data":"bb8a8cd5177db1f978c5107ffab525febb76dde1b8456020773a6655831d9949"}
Nov 03 07:16:33.877872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.877846    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22" event={"ID":"367d6236-a725-4e4d-a86d-68980e9c2672","Type":"ContainerDied","Data":"38b898674a98c6d7c7c96ef5d678d8876e18bb7987a957425ab2da4fbaae5629"}
Nov 03 07:16:33.877872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.877865    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"
Nov 03 07:16:33.878095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:33.877886    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38b898674a98c6d7c7c96ef5d678d8876e18bb7987a957425ab2da4fbaae5629"
Nov 03 07:16:34.759105 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:16:34.759071    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:16:35.032046 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.032023    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:35.133961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.133933    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7d54c2-c364-4b6c-8eb6-283b277046d1-kube-api-access-tz4bh\" (UniqueName: \"kubernetes.io/projected/4f7d54c2-c364-4b6c-8eb6-283b277046d1-kube-api-access-tz4bh\") pod \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") "
Nov 03 07:16:35.134182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.134041    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-openshift-ca\") pod \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") "
Nov 03 07:16:35.134263 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.134213    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-xks-ca\") pod \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\" (UID: \"4f7d54c2-c364-4b6c-8eb6-283b277046d1\") "
Nov 03 07:16:35.134423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.134398    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-openshift-ca" pod "4f7d54c2-c364-4b6c-8eb6-283b277046d1" (UID: "4f7d54c2-c364-4b6c-8eb6-283b277046d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:35.134644 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.134628    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:35.136216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.136192    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7d54c2-c364-4b6c-8eb6-283b277046d1-kube-api-access-tz4bh" pod "4f7d54c2-c364-4b6c-8eb6-283b277046d1" (UID: "4f7d54c2-c364-4b6c-8eb6-283b277046d1"). InnerVolumeSpecName "kube-api-access-tz4bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:35.136361 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.136345    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-xks-ca" pod "4f7d54c2-c364-4b6c-8eb6-283b277046d1" (UID: "4f7d54c2-c364-4b6c-8eb6-283b277046d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:35.235984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.235952    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz4bh\" (UniqueName: \"kubernetes.io/projected/4f7d54c2-c364-4b6c-8eb6-283b277046d1-kube-api-access-tz4bh\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:35.235984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.235979    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7d54c2-c364-4b6c-8eb6-283b277046d1-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:35.840253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.840224    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"]
Nov 03 07:16:35.840804 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.840790    4470 cpu_manager.go:414] "containerMap: removing container" podUID="367d6236-a725-4e4d-a86d-68980e9c2672" containerName="revoke-keys"
Nov 03 07:16:35.840843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.840806    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367d6236-a725-4e4d-a86d-68980e9c2672" containerName="revoke-keys"
Nov 03 07:16:35.840843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.840818    4470 cpu_manager.go:414] "containerMap: removing container" podUID="4f7d54c2-c364-4b6c-8eb6-283b277046d1" containerName="revoke-keys"
Nov 03 07:16:35.840843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.840823    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7d54c2-c364-4b6c-8eb6-283b277046d1" containerName="revoke-keys"
Nov 03 07:16:35.840944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.840934    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="367d6236-a725-4e4d-a86d-68980e9c2672" containerName="revoke-keys"
Nov 03 07:16:35.840977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.840948    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7d54c2-c364-4b6c-8eb6-283b277046d1" containerName="revoke-keys"
Nov 03 07:16:35.844461 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.844441    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:35.892256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.892228    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"
Nov 03 07:16:35.892424 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.892260    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb" event={"ID":"4f7d54c2-c364-4b6c-8eb6-283b277046d1","Type":"ContainerDied","Data":"838da82e5d572d5a4b681cc4dfce0513606a47ecf8dfd60936d3487fe658a049"}
Nov 03 07:16:35.892424 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.892316    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="838da82e5d572d5a4b681cc4dfce0513606a47ecf8dfd60936d3487fe658a049"
Nov 03 07:16:35.905223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.905193    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"]
Nov 03 07:16:35.945014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.944985    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0aadb8a-xnsz4\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:35.945335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.945199    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc69v\" (UniqueName: \"kubernetes.io/projected/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-kube-api-access-tc69v\") pod \"maas-api-revoke-sub-d0aadb8a-xnsz4\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:35.945335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:35.945305    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0aadb8a-xnsz4\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:36.046053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:36.046020    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc69v\" (UniqueName: \"kubernetes.io/projected/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-kube-api-access-tc69v\") pod \"maas-api-revoke-sub-d0aadb8a-xnsz4\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:36.046564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:36.046062    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0aadb8a-xnsz4\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:36.046564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:36.046165    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0aadb8a-xnsz4\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:36.046809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:36.046790    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0aadb8a-xnsz4\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:36.048687 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:36.048664    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0aadb8a-xnsz4\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:36.111278 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:36.111214    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc69v\" (UniqueName: \"kubernetes.io/projected/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-kube-api-access-tc69v\") pod \"maas-api-revoke-sub-d0aadb8a-xnsz4\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:36.158075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:36.158047    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:36.541450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:36.541420    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"]
Nov 03 07:16:36.541863 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:36.541841    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e9f928_ba59_4807_90a2_a02c4a9b26f2.slice/crio-9f6e4f9e4c4995e190b69309b3c266f15f6e7b9ec0eb1272550653a79799b8a3 WatchSource:0}: Error finding container 9f6e4f9e4c4995e190b69309b3c266f15f6e7b9ec0eb1272550653a79799b8a3: Status 404 returned error can't find the container with id 9f6e4f9e4c4995e190b69309b3c266f15f6e7b9ec0eb1272550653a79799b8a3
Nov 03 07:16:36.899272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:36.899241    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4" event={"ID":"e9e9f928-ba59-4807-90a2-a02c4a9b26f2","Type":"ContainerStarted","Data":"3ce29e94d1273d082581a33968d8b09f984bef08ecf76d1ea72f3654def68ce1"}
Nov 03 07:16:36.899418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:36.899298    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4" event={"ID":"e9e9f928-ba59-4807-90a2-a02c4a9b26f2","Type":"ContainerStarted","Data":"9f6e4f9e4c4995e190b69309b3c266f15f6e7b9ec0eb1272550653a79799b8a3"}
Nov 03 07:16:37.019207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:37.019153    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4" podStartSLOduration=2.019123401 podStartE2EDuration="2.019123401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:37.018448892 +0000 UTC m=+1049.350817917" watchObservedRunningTime="2026-11-03 07:16:37.019123401 +0000 UTC m=+1049.351492395"
Nov 03 07:16:37.911680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:37.911632    4470 generic.go:356] "Generic (PLEG): container finished" podID="e9e9f928-ba59-4807-90a2-a02c4a9b26f2" containerID="3ce29e94d1273d082581a33968d8b09f984bef08ecf76d1ea72f3654def68ce1" exitCode=0
Nov 03 07:16:37.912221 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:37.911724    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4" event={"ID":"e9e9f928-ba59-4807-90a2-a02c4a9b26f2","Type":"ContainerDied","Data":"3ce29e94d1273d082581a33968d8b09f984bef08ecf76d1ea72f3654def68ce1"}
Nov 03 07:16:39.070649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.070625    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:39.084984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.084963    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-openshift-ca\") pod \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") "
Nov 03 07:16:39.085362 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.085338    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-openshift-ca" pod "e9e9f928-ba59-4807-90a2-a02c4a9b26f2" (UID: "e9e9f928-ba59-4807-90a2-a02c4a9b26f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:39.085459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.085444    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:39.186337 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.186267    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-kube-api-access-tc69v\" (UniqueName: \"kubernetes.io/projected/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-kube-api-access-tc69v\") pod \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") "
Nov 03 07:16:39.186337 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.186329    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-xks-ca\") pod \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\" (UID: \"e9e9f928-ba59-4807-90a2-a02c4a9b26f2\") "
Nov 03 07:16:39.188546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.188516    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-xks-ca" pod "e9e9f928-ba59-4807-90a2-a02c4a9b26f2" (UID: "e9e9f928-ba59-4807-90a2-a02c4a9b26f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:39.188660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.188641    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-kube-api-access-tc69v" pod "e9e9f928-ba59-4807-90a2-a02c4a9b26f2" (UID: "e9e9f928-ba59-4807-90a2-a02c4a9b26f2"). InnerVolumeSpecName "kube-api-access-tc69v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:39.287974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.287942    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:39.287974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.287970    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc69v\" (UniqueName: \"kubernetes.io/projected/e9e9f928-ba59-4807-90a2-a02c4a9b26f2-kube-api-access-tc69v\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:39.925742 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.925714    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"
Nov 03 07:16:39.925878 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.925714    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4" event={"ID":"e9e9f928-ba59-4807-90a2-a02c4a9b26f2","Type":"ContainerDied","Data":"9f6e4f9e4c4995e190b69309b3c266f15f6e7b9ec0eb1272550653a79799b8a3"}
Nov 03 07:16:39.925878 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:39.925835    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f6e4f9e4c4995e190b69309b3c266f15f6e7b9ec0eb1272550653a79799b8a3"
Nov 03 07:16:41.543189 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.543153    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"]
Nov 03 07:16:41.543724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.543709    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e9e9f928-ba59-4807-90a2-a02c4a9b26f2" containerName="revoke-keys"
Nov 03 07:16:41.543724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.543724    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e9f928-ba59-4807-90a2-a02c4a9b26f2" containerName="revoke-keys"
Nov 03 07:16:41.543835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.543815    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e9f928-ba59-4807-90a2-a02c4a9b26f2" containerName="revoke-keys"
Nov 03 07:16:41.547394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.547378    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:41.607963 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.607931    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"]
Nov 03 07:16:41.614772 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.614736    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03eecb5b-5dsgc\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:41.614928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.614911    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcldf\" (UniqueName: \"kubernetes.io/projected/11d11fe9-d84b-4ec3-b68f-8e199d12d417-kube-api-access-qcldf\") pod \"maas-api-revoke-sub-03eecb5b-5dsgc\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:41.614992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.614949    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03eecb5b-5dsgc\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:41.654513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.654481    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:16:41.654685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.654669    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:16:41.654901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.654890    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:16:41.654938 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.654901    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:16:41.716118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.716086    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03eecb5b-5dsgc\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:41.716289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.716208    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcldf\" (UniqueName: \"kubernetes.io/projected/11d11fe9-d84b-4ec3-b68f-8e199d12d417-kube-api-access-qcldf\") pod \"maas-api-revoke-sub-03eecb5b-5dsgc\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:41.716289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.716237    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03eecb5b-5dsgc\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:41.716771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.716752    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03eecb5b-5dsgc\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:41.718721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.718703    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03eecb5b-5dsgc\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:41.780589 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.780558    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcldf\" (UniqueName: \"kubernetes.io/projected/11d11fe9-d84b-4ec3-b68f-8e199d12d417-kube-api-access-qcldf\") pod \"maas-api-revoke-sub-03eecb5b-5dsgc\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:41.860349 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:41.860314    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:42.240903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.240876    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"]
Nov 03 07:16:42.241716 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:42.241693    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d11fe9_d84b_4ec3_b68f_8e199d12d417.slice/crio-966b5344a0f5e0628dd74768e3d4cbb1069a30303d6b4998e4bb24e77a7f78d0 WatchSource:0}: Error finding container 966b5344a0f5e0628dd74768e3d4cbb1069a30303d6b4998e4bb24e77a7f78d0: Status 404 returned error can't find the container with id 966b5344a0f5e0628dd74768e3d4cbb1069a30303d6b4998e4bb24e77a7f78d0
Nov 03 07:16:42.596379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.596347    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"]
Nov 03 07:16:42.600317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.600300    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:42.625634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.625584    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5088c2d-brfc5\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:42.625795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.625701    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5088c2d-brfc5\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:42.625850 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.625794    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shxlm\" (UniqueName: \"kubernetes.io/projected/0f8b46e6-f4f1-4d63-9414-43d1933129f6-kube-api-access-shxlm\") pod \"maas-api-revoke-sub-b5088c2d-brfc5\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:42.659452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.659422    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"]
Nov 03 07:16:42.727655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.727619    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5088c2d-brfc5\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:42.727851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.727705    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5088c2d-brfc5\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:42.727851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.727827    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shxlm\" (UniqueName: \"kubernetes.io/projected/0f8b46e6-f4f1-4d63-9414-43d1933129f6-kube-api-access-shxlm\") pod \"maas-api-revoke-sub-b5088c2d-brfc5\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:42.728460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.728435    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5088c2d-brfc5\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:42.730387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.730357    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5088c2d-brfc5\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:42.793473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.793442    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shxlm\" (UniqueName: \"kubernetes.io/projected/0f8b46e6-f4f1-4d63-9414-43d1933129f6-kube-api-access-shxlm\") pod \"maas-api-revoke-sub-b5088c2d-brfc5\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:42.913480 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.913444    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:42.947576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.947529    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc" event={"ID":"11d11fe9-d84b-4ec3-b68f-8e199d12d417","Type":"ContainerStarted","Data":"1516d0835e9c35de25e110df2802cbb29452448d237ab191c739ffdecc9a81f2"}
Nov 03 07:16:42.947725 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:42.947614    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc" event={"ID":"11d11fe9-d84b-4ec3-b68f-8e199d12d417","Type":"ContainerStarted","Data":"966b5344a0f5e0628dd74768e3d4cbb1069a30303d6b4998e4bb24e77a7f78d0"}
Nov 03 07:16:43.070604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:43.070553    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc" podStartSLOduration=2.07053882 podStartE2EDuration="2.07053882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:43.068174462 +0000 UTC m=+1055.400543480" watchObservedRunningTime="2026-11-03 07:16:43.07053882 +0000 UTC m=+1055.402907814"
Nov 03 07:16:43.293112 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:43.293077    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"]
Nov 03 07:16:43.293722 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:43.293697    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8b46e6_f4f1_4d63_9414_43d1933129f6.slice/crio-30d375b6eab1395e781a84aa5bdd37c6a815321662ae48fd9ed8be84c56d27c1 WatchSource:0}: Error finding container 30d375b6eab1395e781a84aa5bdd37c6a815321662ae48fd9ed8be84c56d27c1: Status 404 returned error can't find the container with id 30d375b6eab1395e781a84aa5bdd37c6a815321662ae48fd9ed8be84c56d27c1
Nov 03 07:16:43.956950 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:43.956907    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5" event={"ID":"0f8b46e6-f4f1-4d63-9414-43d1933129f6","Type":"ContainerStarted","Data":"2c991703d9a25f0f5a0488bd3ade7e52d95cb227b3c2814ecfb70eaa6324d234"}
Nov 03 07:16:43.957440 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:43.956974    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5" event={"ID":"0f8b46e6-f4f1-4d63-9414-43d1933129f6","Type":"ContainerStarted","Data":"30d375b6eab1395e781a84aa5bdd37c6a815321662ae48fd9ed8be84c56d27c1"}
Nov 03 07:16:43.958485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:43.958448    4470 generic.go:356] "Generic (PLEG): container finished" podID="11d11fe9-d84b-4ec3-b68f-8e199d12d417" containerID="1516d0835e9c35de25e110df2802cbb29452448d237ab191c739ffdecc9a81f2" exitCode=0
Nov 03 07:16:43.958586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:43.958527    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc" event={"ID":"11d11fe9-d84b-4ec3-b68f-8e199d12d417","Type":"ContainerDied","Data":"1516d0835e9c35de25e110df2802cbb29452448d237ab191c739ffdecc9a81f2"}
Nov 03 07:16:44.076941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.076892    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5" podStartSLOduration=2.076877219 podStartE2EDuration="2.076877219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:44.076051138 +0000 UTC m=+1056.408420133" watchObservedRunningTime="2026-11-03 07:16:44.076877219 +0000 UTC m=+1056.409246214"
Nov 03 07:16:44.634325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.634290    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"]
Nov 03 07:16:44.638243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.638228    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:44.696851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.696815    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"]
Nov 03 07:16:44.747437 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.747397    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c857612-85dnk\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:44.747604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.747447    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfrzf\" (UniqueName: \"kubernetes.io/projected/fb12da53-2b74-4a64-aa6b-b1b5641e9542-kube-api-access-hfrzf\") pod \"maas-api-revoke-sub-1c857612-85dnk\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:44.747604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.747470    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c857612-85dnk\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:44.849273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.849232    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c857612-85dnk\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:44.849473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.849299    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfrzf\" (UniqueName: \"kubernetes.io/projected/fb12da53-2b74-4a64-aa6b-b1b5641e9542-kube-api-access-hfrzf\") pod \"maas-api-revoke-sub-1c857612-85dnk\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:44.849473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.849333    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c857612-85dnk\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:44.849903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.849871    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c857612-85dnk\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:44.851959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.851931    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c857612-85dnk\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:44.916953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.916881    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfrzf\" (UniqueName: \"kubernetes.io/projected/fb12da53-2b74-4a64-aa6b-b1b5641e9542-kube-api-access-hfrzf\") pod \"maas-api-revoke-sub-1c857612-85dnk\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:44.953743 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.953671    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:44.965950 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.965923    4470 generic.go:356] "Generic (PLEG): container finished" podID="0f8b46e6-f4f1-4d63-9414-43d1933129f6" containerID="2c991703d9a25f0f5a0488bd3ade7e52d95cb227b3c2814ecfb70eaa6324d234" exitCode=0
Nov 03 07:16:44.966369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:44.966004    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5" event={"ID":"0f8b46e6-f4f1-4d63-9414-43d1933129f6","Type":"ContainerDied","Data":"2c991703d9a25f0f5a0488bd3ade7e52d95cb227b3c2814ecfb70eaa6324d234"}
Nov 03 07:16:45.156637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.156606    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:45.254855 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.254773    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d11fe9-d84b-4ec3-b68f-8e199d12d417-kube-api-access-qcldf\" (UniqueName: \"kubernetes.io/projected/11d11fe9-d84b-4ec3-b68f-8e199d12d417-kube-api-access-qcldf\") pod \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") "
Nov 03 07:16:45.254992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.254879    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-openshift-ca\") pod \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") "
Nov 03 07:16:45.254992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.254940    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-xks-ca\") pod \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\" (UID: \"11d11fe9-d84b-4ec3-b68f-8e199d12d417\") "
Nov 03 07:16:45.255252 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.255229    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-openshift-ca" pod "11d11fe9-d84b-4ec3-b68f-8e199d12d417" (UID: "11d11fe9-d84b-4ec3-b68f-8e199d12d417"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:45.257921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.257891    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-xks-ca" pod "11d11fe9-d84b-4ec3-b68f-8e199d12d417" (UID: "11d11fe9-d84b-4ec3-b68f-8e199d12d417"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:45.258087 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.258022    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d11fe9-d84b-4ec3-b68f-8e199d12d417-kube-api-access-qcldf" pod "11d11fe9-d84b-4ec3-b68f-8e199d12d417" (UID: "11d11fe9-d84b-4ec3-b68f-8e199d12d417"). InnerVolumeSpecName "kube-api-access-qcldf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:45.338831 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.338801    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"]
Nov 03 07:16:45.339050 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:45.339021    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb12da53_2b74_4a64_aa6b_b1b5641e9542.slice/crio-844bddccc776604aabad0a29be478ffdca89d958f578b61123df4c2fe9441890 WatchSource:0}: Error finding container 844bddccc776604aabad0a29be478ffdca89d958f578b61123df4c2fe9441890: Status 404 returned error can't find the container with id 844bddccc776604aabad0a29be478ffdca89d958f578b61123df4c2fe9441890
Nov 03 07:16:45.356739 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.356712    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:45.356883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.356744    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d11fe9-d84b-4ec3-b68f-8e199d12d417-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:45.356883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.356758    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcldf\" (UniqueName: \"kubernetes.io/projected/11d11fe9-d84b-4ec3-b68f-8e199d12d417-kube-api-access-qcldf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:45.973600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.973569    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"
Nov 03 07:16:45.974117 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.973614    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc" event={"ID":"11d11fe9-d84b-4ec3-b68f-8e199d12d417","Type":"ContainerDied","Data":"966b5344a0f5e0628dd74768e3d4cbb1069a30303d6b4998e4bb24e77a7f78d0"}
Nov 03 07:16:45.974117 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.973665    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="966b5344a0f5e0628dd74768e3d4cbb1069a30303d6b4998e4bb24e77a7f78d0"
Nov 03 07:16:45.975470 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.975439    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk" event={"ID":"fb12da53-2b74-4a64-aa6b-b1b5641e9542","Type":"ContainerStarted","Data":"50ccb3ad3ccfc2d76a5355f973d193646c197b3c2f37bb4469272bb9277c4547"}
Nov 03 07:16:45.975851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:45.975499    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk" event={"ID":"fb12da53-2b74-4a64-aa6b-b1b5641e9542","Type":"ContainerStarted","Data":"844bddccc776604aabad0a29be478ffdca89d958f578b61123df4c2fe9441890"}
Nov 03 07:16:46.096027 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.095969    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk" podStartSLOduration=2.095950338 podStartE2EDuration="2.095950338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:46.095899524 +0000 UTC m=+1058.428268515" watchObservedRunningTime="2026-11-03 07:16:46.095950338 +0000 UTC m=+1058.428319330"
Nov 03 07:16:46.139099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.139013    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:46.164747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.164718    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-openshift-ca\") pod \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") "
Nov 03 07:16:46.164915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.164855    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8b46e6-f4f1-4d63-9414-43d1933129f6-kube-api-access-shxlm\" (UniqueName: \"kubernetes.io/projected/0f8b46e6-f4f1-4d63-9414-43d1933129f6-kube-api-access-shxlm\") pod \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") "
Nov 03 07:16:46.164915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.164891    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-xks-ca\") pod \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\" (UID: \"0f8b46e6-f4f1-4d63-9414-43d1933129f6\") "
Nov 03 07:16:46.165149 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.165107    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-openshift-ca" pod "0f8b46e6-f4f1-4d63-9414-43d1933129f6" (UID: "0f8b46e6-f4f1-4d63-9414-43d1933129f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:46.165339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.165324    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:46.167106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.167071    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-xks-ca" pod "0f8b46e6-f4f1-4d63-9414-43d1933129f6" (UID: "0f8b46e6-f4f1-4d63-9414-43d1933129f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:46.167106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.167074    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8b46e6-f4f1-4d63-9414-43d1933129f6-kube-api-access-shxlm" pod "0f8b46e6-f4f1-4d63-9414-43d1933129f6" (UID: "0f8b46e6-f4f1-4d63-9414-43d1933129f6"). InnerVolumeSpecName "kube-api-access-shxlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:46.268526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.268210    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shxlm\" (UniqueName: \"kubernetes.io/projected/0f8b46e6-f4f1-4d63-9414-43d1933129f6-kube-api-access-shxlm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:46.268526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.268265    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8b46e6-f4f1-4d63-9414-43d1933129f6-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:46.983310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.983278    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"
Nov 03 07:16:46.983789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.983310    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5" event={"ID":"0f8b46e6-f4f1-4d63-9414-43d1933129f6","Type":"ContainerDied","Data":"30d375b6eab1395e781a84aa5bdd37c6a815321662ae48fd9ed8be84c56d27c1"}
Nov 03 07:16:46.983789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.983362    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30d375b6eab1395e781a84aa5bdd37c6a815321662ae48fd9ed8be84c56d27c1"
Nov 03 07:16:46.985005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.984981    4470 generic.go:356] "Generic (PLEG): container finished" podID="fb12da53-2b74-4a64-aa6b-b1b5641e9542" containerID="50ccb3ad3ccfc2d76a5355f973d193646c197b3c2f37bb4469272bb9277c4547" exitCode=0
Nov 03 07:16:46.985172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:46.985066    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk" event={"ID":"fb12da53-2b74-4a64-aa6b-b1b5641e9542","Type":"ContainerDied","Data":"50ccb3ad3ccfc2d76a5355f973d193646c197b3c2f37bb4469272bb9277c4547"}
Nov 03 07:16:47.542934 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.542901    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"]
Nov 03 07:16:47.543466 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.543450    4470 cpu_manager.go:414] "containerMap: removing container" podUID="0f8b46e6-f4f1-4d63-9414-43d1933129f6" containerName="revoke-keys"
Nov 03 07:16:47.543466 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.543466    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8b46e6-f4f1-4d63-9414-43d1933129f6" containerName="revoke-keys"
Nov 03 07:16:47.543599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.543480    4470 cpu_manager.go:414] "containerMap: removing container" podUID="11d11fe9-d84b-4ec3-b68f-8e199d12d417" containerName="revoke-keys"
Nov 03 07:16:47.543599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.543486    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d11fe9-d84b-4ec3-b68f-8e199d12d417" containerName="revoke-keys"
Nov 03 07:16:47.543599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.543573    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8b46e6-f4f1-4d63-9414-43d1933129f6" containerName="revoke-keys"
Nov 03 07:16:47.543599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.543582    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d11fe9-d84b-4ec3-b68f-8e199d12d417" containerName="revoke-keys"
Nov 03 07:16:47.546947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.546921    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:47.582394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.582361    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9hlf\" (UniqueName: \"kubernetes.io/projected/ae326739-3144-425d-86c9-e1502b925c3e-kube-api-access-s9hlf\") pod \"maas-api-revoke-sub-7dacfa8f-vrz6w\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:47.582572 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.582510    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dacfa8f-vrz6w\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:47.582572 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.582565    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dacfa8f-vrz6w\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:47.606414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.606383    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"]
Nov 03 07:16:47.684078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.684049    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9hlf\" (UniqueName: \"kubernetes.io/projected/ae326739-3144-425d-86c9-e1502b925c3e-kube-api-access-s9hlf\") pod \"maas-api-revoke-sub-7dacfa8f-vrz6w\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:47.684302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.684121    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dacfa8f-vrz6w\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:47.684302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.684168    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dacfa8f-vrz6w\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:47.684753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.684733    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dacfa8f-vrz6w\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:47.686626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.686605    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dacfa8f-vrz6w\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:47.751063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.751028    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9hlf\" (UniqueName: \"kubernetes.io/projected/ae326739-3144-425d-86c9-e1502b925c3e-kube-api-access-s9hlf\") pod \"maas-api-revoke-sub-7dacfa8f-vrz6w\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:47.862198 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:47.862160    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:48.145612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.145586    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:48.189521 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.189490    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-openshift-ca\") pod \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") "
Nov 03 07:16:48.189697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.189587    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb12da53-2b74-4a64-aa6b-b1b5641e9542-kube-api-access-hfrzf\" (UniqueName: \"kubernetes.io/projected/fb12da53-2b74-4a64-aa6b-b1b5641e9542-kube-api-access-hfrzf\") pod \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") "
Nov 03 07:16:48.189765 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.189712    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-xks-ca\") pod \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\" (UID: \"fb12da53-2b74-4a64-aa6b-b1b5641e9542\") "
Nov 03 07:16:48.189931 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.189903    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-openshift-ca" pod "fb12da53-2b74-4a64-aa6b-b1b5641e9542" (UID: "fb12da53-2b74-4a64-aa6b-b1b5641e9542"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:48.190239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.190215    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:48.192106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.192076    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb12da53-2b74-4a64-aa6b-b1b5641e9542-kube-api-access-hfrzf" pod "fb12da53-2b74-4a64-aa6b-b1b5641e9542" (UID: "fb12da53-2b74-4a64-aa6b-b1b5641e9542"). InnerVolumeSpecName "kube-api-access-hfrzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:48.192227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.192152    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-xks-ca" pod "fb12da53-2b74-4a64-aa6b-b1b5641e9542" (UID: "fb12da53-2b74-4a64-aa6b-b1b5641e9542"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:48.241066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.241030    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"]
Nov 03 07:16:48.242056 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:48.242031    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae326739_3144_425d_86c9_e1502b925c3e.slice/crio-ccc4b9d38a724ec2d935f4653013b413da05411fb2c89d663cc35bc8873f52aa WatchSource:0}: Error finding container ccc4b9d38a724ec2d935f4653013b413da05411fb2c89d663cc35bc8873f52aa: Status 404 returned error can't find the container with id ccc4b9d38a724ec2d935f4653013b413da05411fb2c89d663cc35bc8873f52aa
Nov 03 07:16:48.291531 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.291497    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfrzf\" (UniqueName: \"kubernetes.io/projected/fb12da53-2b74-4a64-aa6b-b1b5641e9542-kube-api-access-hfrzf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:48.291670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.291534    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb12da53-2b74-4a64-aa6b-b1b5641e9542-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:48.999469 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.999431    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w" event={"ID":"ae326739-3144-425d-86c9-e1502b925c3e","Type":"ContainerStarted","Data":"5d5b1cedeb22f29b8f7d31b5729b66ea919d7c4647dd0b2069d904f7b79d8d43"}
Nov 03 07:16:48.999671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:48.999496    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w" event={"ID":"ae326739-3144-425d-86c9-e1502b925c3e","Type":"ContainerStarted","Data":"ccc4b9d38a724ec2d935f4653013b413da05411fb2c89d663cc35bc8873f52aa"}
Nov 03 07:16:49.000925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:49.000905    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"
Nov 03 07:16:49.000925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:49.000903    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk" event={"ID":"fb12da53-2b74-4a64-aa6b-b1b5641e9542","Type":"ContainerDied","Data":"844bddccc776604aabad0a29be478ffdca89d958f578b61123df4c2fe9441890"}
Nov 03 07:16:49.001088 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:49.000952    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="844bddccc776604aabad0a29be478ffdca89d958f578b61123df4c2fe9441890"
Nov 03 07:16:49.119296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:49.119235    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w" podStartSLOduration=2.119221122 podStartE2EDuration="2.119221122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:49.118335035 +0000 UTC m=+1061.450704020" watchObservedRunningTime="2026-11-03 07:16:49.119221122 +0000 UTC m=+1061.451590116"
Nov 03 07:16:50.009055 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:50.009014    4470 generic.go:356] "Generic (PLEG): container finished" podID="ae326739-3144-425d-86c9-e1502b925c3e" containerID="5d5b1cedeb22f29b8f7d31b5729b66ea919d7c4647dd0b2069d904f7b79d8d43" exitCode=0
Nov 03 07:16:50.009530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:50.009097    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w" event={"ID":"ae326739-3144-425d-86c9-e1502b925c3e","Type":"ContainerDied","Data":"5d5b1cedeb22f29b8f7d31b5729b66ea919d7c4647dd0b2069d904f7b79d8d43"}
Nov 03 07:16:51.164062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.164038    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:51.225573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.225547    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-xks-ca\") pod \"ae326739-3144-425d-86c9-e1502b925c3e\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") "
Nov 03 07:16:51.225752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.225625    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae326739-3144-425d-86c9-e1502b925c3e-kube-api-access-s9hlf\" (UniqueName: \"kubernetes.io/projected/ae326739-3144-425d-86c9-e1502b925c3e-kube-api-access-s9hlf\") pod \"ae326739-3144-425d-86c9-e1502b925c3e\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") "
Nov 03 07:16:51.225752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.225647    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-openshift-ca\") pod \"ae326739-3144-425d-86c9-e1502b925c3e\" (UID: \"ae326739-3144-425d-86c9-e1502b925c3e\") "
Nov 03 07:16:51.226050 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.226028    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-openshift-ca" pod "ae326739-3144-425d-86c9-e1502b925c3e" (UID: "ae326739-3144-425d-86c9-e1502b925c3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:51.227900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.227874    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-xks-ca" pod "ae326739-3144-425d-86c9-e1502b925c3e" (UID: "ae326739-3144-425d-86c9-e1502b925c3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:51.227900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.227877    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae326739-3144-425d-86c9-e1502b925c3e-kube-api-access-s9hlf" pod "ae326739-3144-425d-86c9-e1502b925c3e" (UID: "ae326739-3144-425d-86c9-e1502b925c3e"). InnerVolumeSpecName "kube-api-access-s9hlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:51.332747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.332643    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:51.332747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.332697    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9hlf\" (UniqueName: \"kubernetes.io/projected/ae326739-3144-425d-86c9-e1502b925c3e-kube-api-access-s9hlf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:51.332747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.332715    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae326739-3144-425d-86c9-e1502b925c3e-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:51.925361 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.925330    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"]
Nov 03 07:16:51.926003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.925981    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ae326739-3144-425d-86c9-e1502b925c3e" containerName="revoke-keys"
Nov 03 07:16:51.926003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.926002    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae326739-3144-425d-86c9-e1502b925c3e" containerName="revoke-keys"
Nov 03 07:16:51.926215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.926021    4470 cpu_manager.go:414] "containerMap: removing container" podUID="fb12da53-2b74-4a64-aa6b-b1b5641e9542" containerName="revoke-keys"
Nov 03 07:16:51.926215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.926029    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb12da53-2b74-4a64-aa6b-b1b5641e9542" containerName="revoke-keys"
Nov 03 07:16:51.926215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.926170    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae326739-3144-425d-86c9-e1502b925c3e" containerName="revoke-keys"
Nov 03 07:16:51.926215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.926186    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb12da53-2b74-4a64-aa6b-b1b5641e9542" containerName="revoke-keys"
Nov 03 07:16:51.929807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.929788    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:51.990946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:51.990913    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"]
Nov 03 07:16:52.024868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.024836    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"
Nov 03 07:16:52.025024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.024861    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w" event={"ID":"ae326739-3144-425d-86c9-e1502b925c3e","Type":"ContainerDied","Data":"ccc4b9d38a724ec2d935f4653013b413da05411fb2c89d663cc35bc8873f52aa"}
Nov 03 07:16:52.025024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.024913    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccc4b9d38a724ec2d935f4653013b413da05411fb2c89d663cc35bc8873f52aa"
Nov 03 07:16:52.040338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.040314    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snt8\" (UniqueName: \"kubernetes.io/projected/522620d2-f7b5-4eca-b973-60ec250831f4-kube-api-access-4snt8\") pod \"maas-api-revoke-sub-affe0c8f-5l8ft\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:52.040454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.040347    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affe0c8f-5l8ft\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:52.040511 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.040464    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affe0c8f-5l8ft\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:52.141464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.141386    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snt8\" (UniqueName: \"kubernetes.io/projected/522620d2-f7b5-4eca-b973-60ec250831f4-kube-api-access-4snt8\") pod \"maas-api-revoke-sub-affe0c8f-5l8ft\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:52.141464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.141423    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affe0c8f-5l8ft\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:52.141649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.141490    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affe0c8f-5l8ft\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:52.142220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.142201    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affe0c8f-5l8ft\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:52.143950 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.143929    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affe0c8f-5l8ft\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:52.211430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.211399    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snt8\" (UniqueName: \"kubernetes.io/projected/522620d2-f7b5-4eca-b973-60ec250831f4-kube-api-access-4snt8\") pod \"maas-api-revoke-sub-affe0c8f-5l8ft\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:52.242900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.242872    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:52.365242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.365213    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"]
Nov 03 07:16:52.370220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.370204    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:52.431038 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.430965    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"]
Nov 03 07:16:52.444601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.444571    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2vzq\" (UniqueName: \"kubernetes.io/projected/3a33ee05-4ed7-4a92-a154-6f659e8854ae-kube-api-access-m2vzq\") pod \"maas-api-revoke-sub-4ba24433-hp2dx\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:52.444750 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.444625    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba24433-hp2dx\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:52.444750 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.444655    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba24433-hp2dx\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:52.545646 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.545612    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2vzq\" (UniqueName: \"kubernetes.io/projected/3a33ee05-4ed7-4a92-a154-6f659e8854ae-kube-api-access-m2vzq\") pod \"maas-api-revoke-sub-4ba24433-hp2dx\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:52.545796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.545662    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba24433-hp2dx\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:52.545796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.545688    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba24433-hp2dx\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:52.546348 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.546331    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba24433-hp2dx\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:52.547994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.547965    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba24433-hp2dx\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:52.612019 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.611985    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2vzq\" (UniqueName: \"kubernetes.io/projected/3a33ee05-4ed7-4a92-a154-6f659e8854ae-kube-api-access-m2vzq\") pod \"maas-api-revoke-sub-4ba24433-hp2dx\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:52.634508 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.634482    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"]
Nov 03 07:16:52.634994 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:52.634969    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522620d2_f7b5_4eca_b973_60ec250831f4.slice/crio-0f5b46dbb542288214c237f7a54796300719c8a3c30204af5995a46ea0ea58c5 WatchSource:0}: Error finding container 0f5b46dbb542288214c237f7a54796300719c8a3c30204af5995a46ea0ea58c5: Status 404 returned error can't find the container with id 0f5b46dbb542288214c237f7a54796300719c8a3c30204af5995a46ea0ea58c5
Nov 03 07:16:52.683951 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:52.683894    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:53.032473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.032435    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft" event={"ID":"522620d2-f7b5-4eca-b973-60ec250831f4","Type":"ContainerStarted","Data":"340f8d2523b839898019d84ec470fa0cca2e01796e6b3135548d04b2ac436f86"}
Nov 03 07:16:53.032665 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.032490    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft" event={"ID":"522620d2-f7b5-4eca-b973-60ec250831f4","Type":"ContainerStarted","Data":"0f5b46dbb542288214c237f7a54796300719c8a3c30204af5995a46ea0ea58c5"}
Nov 03 07:16:53.062165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.062122    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"]
Nov 03 07:16:53.062565 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:53.062522    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a33ee05_4ed7_4a92_a154_6f659e8854ae.slice/crio-0f753002fd4f94e9de33a757ac6e8ac4b5064fa7884e732213605528adecdecd WatchSource:0}: Error finding container 0f753002fd4f94e9de33a757ac6e8ac4b5064fa7884e732213605528adecdecd: Status 404 returned error can't find the container with id 0f753002fd4f94e9de33a757ac6e8ac4b5064fa7884e732213605528adecdecd
Nov 03 07:16:53.152443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.152395    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft" podStartSLOduration=2.15237664 podStartE2EDuration="2.15237664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:53.152167662 +0000 UTC m=+1065.484536651" watchObservedRunningTime="2026-11-03 07:16:53.15237664 +0000 UTC m=+1065.484745634"
Nov 03 07:16:53.410175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.410123    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"]
Nov 03 07:16:53.414339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.414318    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:53.472780 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.472743    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"]
Nov 03 07:16:53.559010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.558980    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dcd155a-zh7wg\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:53.559175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.559022    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktzp5\" (UniqueName: \"kubernetes.io/projected/15b6fde5-fcde-4f39-9aa6-7821141f37d5-kube-api-access-ktzp5\") pod \"maas-api-revoke-sub-8dcd155a-zh7wg\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:53.559242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.559179    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dcd155a-zh7wg\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:53.660318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.660290    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dcd155a-zh7wg\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:53.660318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.660328    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktzp5\" (UniqueName: \"kubernetes.io/projected/15b6fde5-fcde-4f39-9aa6-7821141f37d5-kube-api-access-ktzp5\") pod \"maas-api-revoke-sub-8dcd155a-zh7wg\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:53.660494 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.660376    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dcd155a-zh7wg\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:53.660955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.660931    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dcd155a-zh7wg\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:53.662773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.662755    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dcd155a-zh7wg\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:53.725541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.725513    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktzp5\" (UniqueName: \"kubernetes.io/projected/15b6fde5-fcde-4f39-9aa6-7821141f37d5-kube-api-access-ktzp5\") pod \"maas-api-revoke-sub-8dcd155a-zh7wg\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:53.727359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:53.727336    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:54.040875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:54.040791    4470 generic.go:356] "Generic (PLEG): container finished" podID="3a33ee05-4ed7-4a92-a154-6f659e8854ae" containerID="ab8283ed7096301da506b0a9cb24cc2321de5e44f2241ffcf2b0de32bf2a8cc3" exitCode=0
Nov 03 07:16:54.040875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:54.040866    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx" event={"ID":"3a33ee05-4ed7-4a92-a154-6f659e8854ae","Type":"ContainerDied","Data":"ab8283ed7096301da506b0a9cb24cc2321de5e44f2241ffcf2b0de32bf2a8cc3"}
Nov 03 07:16:54.041066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:54.040914    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx" event={"ID":"3a33ee05-4ed7-4a92-a154-6f659e8854ae","Type":"ContainerStarted","Data":"0f753002fd4f94e9de33a757ac6e8ac4b5064fa7884e732213605528adecdecd"}
Nov 03 07:16:54.042425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:54.042401    4470 generic.go:356] "Generic (PLEG): container finished" podID="522620d2-f7b5-4eca-b973-60ec250831f4" containerID="340f8d2523b839898019d84ec470fa0cca2e01796e6b3135548d04b2ac436f86" exitCode=0
Nov 03 07:16:54.042542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:54.042458    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft" event={"ID":"522620d2-f7b5-4eca-b973-60ec250831f4","Type":"ContainerDied","Data":"340f8d2523b839898019d84ec470fa0cca2e01796e6b3135548d04b2ac436f86"}
Nov 03 07:16:54.111073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:54.111047    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"]
Nov 03 07:16:54.111779 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:16:54.111752    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b6fde5_fcde_4f39_9aa6_7821141f37d5.slice/crio-3e5126ad5826f5bf056d7c12c67f743096d9337b175e9a6e59012fe7f3676c57 WatchSource:0}: Error finding container 3e5126ad5826f5bf056d7c12c67f743096d9337b175e9a6e59012fe7f3676c57: Status 404 returned error can't find the container with id 3e5126ad5826f5bf056d7c12c67f743096d9337b175e9a6e59012fe7f3676c57
Nov 03 07:16:55.050568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.050531    4470 generic.go:356] "Generic (PLEG): container finished" podID="15b6fde5-fcde-4f39-9aa6-7821141f37d5" containerID="d5f3c232578c6811a2074a8759d9f7be2fe7171b76b3a8a0ad26556b6a013476" exitCode=0
Nov 03 07:16:55.051003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.050613    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg" event={"ID":"15b6fde5-fcde-4f39-9aa6-7821141f37d5","Type":"ContainerDied","Data":"d5f3c232578c6811a2074a8759d9f7be2fe7171b76b3a8a0ad26556b6a013476"}
Nov 03 07:16:55.051003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.050710    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg" event={"ID":"15b6fde5-fcde-4f39-9aa6-7821141f37d5","Type":"ContainerStarted","Data":"3e5126ad5826f5bf056d7c12c67f743096d9337b175e9a6e59012fe7f3676c57"}
Nov 03 07:16:55.206552 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.206485    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:55.245924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.245903    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:55.280067 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.280027    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-openshift-ca\") pod \"522620d2-f7b5-4eca-b973-60ec250831f4\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") "
Nov 03 07:16:55.280067 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.280070    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-xks-ca\") pod \"522620d2-f7b5-4eca-b973-60ec250831f4\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") "
Nov 03 07:16:55.280333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.280121    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522620d2-f7b5-4eca-b973-60ec250831f4-kube-api-access-4snt8\" (UniqueName: \"kubernetes.io/projected/522620d2-f7b5-4eca-b973-60ec250831f4-kube-api-access-4snt8\") pod \"522620d2-f7b5-4eca-b973-60ec250831f4\" (UID: \"522620d2-f7b5-4eca-b973-60ec250831f4\") "
Nov 03 07:16:55.280484 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.280455    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-openshift-ca" pod "522620d2-f7b5-4eca-b973-60ec250831f4" (UID: "522620d2-f7b5-4eca-b973-60ec250831f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:55.282339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.282309    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522620d2-f7b5-4eca-b973-60ec250831f4-kube-api-access-4snt8" pod "522620d2-f7b5-4eca-b973-60ec250831f4" (UID: "522620d2-f7b5-4eca-b973-60ec250831f4"). InnerVolumeSpecName "kube-api-access-4snt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:55.282458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.282344    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-xks-ca" pod "522620d2-f7b5-4eca-b973-60ec250831f4" (UID: "522620d2-f7b5-4eca-b973-60ec250831f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:55.380988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.380955    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-xks-ca\") pod \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") "
Nov 03 07:16:55.381206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.381187    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-openshift-ca\") pod \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") "
Nov 03 07:16:55.381270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.381253    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a33ee05-4ed7-4a92-a154-6f659e8854ae-kube-api-access-m2vzq\" (UniqueName: \"kubernetes.io/projected/3a33ee05-4ed7-4a92-a154-6f659e8854ae-kube-api-access-m2vzq\") pod \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\" (UID: \"3a33ee05-4ed7-4a92-a154-6f659e8854ae\") "
Nov 03 07:16:55.381564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.381538    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-openshift-ca" pod "3a33ee05-4ed7-4a92-a154-6f659e8854ae" (UID: "3a33ee05-4ed7-4a92-a154-6f659e8854ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:55.381759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.381738    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:55.381872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.381763    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522620d2-f7b5-4eca-b973-60ec250831f4-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:55.381872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.381777    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4snt8\" (UniqueName: \"kubernetes.io/projected/522620d2-f7b5-4eca-b973-60ec250831f4-kube-api-access-4snt8\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:55.381872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.381789    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:55.383238 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.383214    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-xks-ca" pod "3a33ee05-4ed7-4a92-a154-6f659e8854ae" (UID: "3a33ee05-4ed7-4a92-a154-6f659e8854ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:55.383436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.383419    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a33ee05-4ed7-4a92-a154-6f659e8854ae-kube-api-access-m2vzq" pod "3a33ee05-4ed7-4a92-a154-6f659e8854ae" (UID: "3a33ee05-4ed7-4a92-a154-6f659e8854ae"). InnerVolumeSpecName "kube-api-access-m2vzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:55.482736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.482644    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2vzq\" (UniqueName: \"kubernetes.io/projected/3a33ee05-4ed7-4a92-a154-6f659e8854ae-kube-api-access-m2vzq\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:55.482736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:55.482673    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a33ee05-4ed7-4a92-a154-6f659e8854ae-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:56.058268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.058229    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx" event={"ID":"3a33ee05-4ed7-4a92-a154-6f659e8854ae","Type":"ContainerDied","Data":"0f753002fd4f94e9de33a757ac6e8ac4b5064fa7884e732213605528adecdecd"}
Nov 03 07:16:56.058268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.058250    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"
Nov 03 07:16:56.058739 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.058288    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f753002fd4f94e9de33a757ac6e8ac4b5064fa7884e732213605528adecdecd"
Nov 03 07:16:56.059634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.059599    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft" event={"ID":"522620d2-f7b5-4eca-b973-60ec250831f4","Type":"ContainerDied","Data":"0f5b46dbb542288214c237f7a54796300719c8a3c30204af5995a46ea0ea58c5"}
Nov 03 07:16:56.059758 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.059657    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5b46dbb542288214c237f7a54796300719c8a3c30204af5995a46ea0ea58c5"
Nov 03 07:16:56.059758 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.059617    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"
Nov 03 07:16:56.203063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.203037    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:56.292321 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.292286    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-xks-ca\") pod \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") "
Nov 03 07:16:56.292532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.292357    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-openshift-ca\") pod \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") "
Nov 03 07:16:56.292532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.292387    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b6fde5-fcde-4f39-9aa6-7821141f37d5-kube-api-access-ktzp5\" (UniqueName: \"kubernetes.io/projected/15b6fde5-fcde-4f39-9aa6-7821141f37d5-kube-api-access-ktzp5\") pod \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\" (UID: \"15b6fde5-fcde-4f39-9aa6-7821141f37d5\") "
Nov 03 07:16:56.292723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.292699    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-openshift-ca" pod "15b6fde5-fcde-4f39-9aa6-7821141f37d5" (UID: "15b6fde5-fcde-4f39-9aa6-7821141f37d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:16:56.294567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.294537    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-xks-ca" pod "15b6fde5-fcde-4f39-9aa6-7821141f37d5" (UID: "15b6fde5-fcde-4f39-9aa6-7821141f37d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:16:56.294680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.294662    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b6fde5-fcde-4f39-9aa6-7821141f37d5-kube-api-access-ktzp5" pod "15b6fde5-fcde-4f39-9aa6-7821141f37d5" (UID: "15b6fde5-fcde-4f39-9aa6-7821141f37d5"). InnerVolumeSpecName "kube-api-access-ktzp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:16:56.394052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.393973    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:56.394052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.394001    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6fde5-fcde-4f39-9aa6-7821141f37d5-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:56.394052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:56.394010    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktzp5\" (UniqueName: \"kubernetes.io/projected/15b6fde5-fcde-4f39-9aa6-7821141f37d5-kube-api-access-ktzp5\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:16:57.067295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.067256    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg" event={"ID":"15b6fde5-fcde-4f39-9aa6-7821141f37d5","Type":"ContainerDied","Data":"3e5126ad5826f5bf056d7c12c67f743096d9337b175e9a6e59012fe7f3676c57"}
Nov 03 07:16:57.067669 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.067310    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e5126ad5826f5bf056d7c12c67f743096d9337b175e9a6e59012fe7f3676c57"
Nov 03 07:16:57.067669 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.067276    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"
Nov 03 07:16:57.592251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.592221    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"]
Nov 03 07:16:57.592751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.592739    4470 cpu_manager.go:414] "containerMap: removing container" podUID="15b6fde5-fcde-4f39-9aa6-7821141f37d5" containerName="revoke-keys"
Nov 03 07:16:57.592790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.592753    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b6fde5-fcde-4f39-9aa6-7821141f37d5" containerName="revoke-keys"
Nov 03 07:16:57.592790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.592763    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3a33ee05-4ed7-4a92-a154-6f659e8854ae" containerName="revoke-keys"
Nov 03 07:16:57.592790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.592769    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a33ee05-4ed7-4a92-a154-6f659e8854ae" containerName="revoke-keys"
Nov 03 07:16:57.592790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.592787    4470 cpu_manager.go:414] "containerMap: removing container" podUID="522620d2-f7b5-4eca-b973-60ec250831f4" containerName="revoke-keys"
Nov 03 07:16:57.592926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.592792    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522620d2-f7b5-4eca-b973-60ec250831f4" containerName="revoke-keys"
Nov 03 07:16:57.592926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.592903    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="522620d2-f7b5-4eca-b973-60ec250831f4" containerName="revoke-keys"
Nov 03 07:16:57.592926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.592919    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b6fde5-fcde-4f39-9aa6-7821141f37d5" containerName="revoke-keys"
Nov 03 07:16:57.593012 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.592928    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a33ee05-4ed7-4a92-a154-6f659e8854ae" containerName="revoke-keys"
Nov 03 07:16:57.596454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.596437    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:16:57.652693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.652661    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:16:57.652900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.652842    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:16:57.656426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.656398    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"]
Nov 03 07:16:57.702357 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.702329    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:16:57.702357 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.702357    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:16:57.708926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.708897    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22df6623-5855-4e65-909a-669d291db280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38bf2752-kd8tt\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:16:57.709073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.709038    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n85z\" (UniqueName: \"kubernetes.io/projected/22df6623-5855-4e65-909a-669d291db280-kube-api-access-5n85z\") pod \"maas-api-revoke-sub-38bf2752-kd8tt\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:16:57.709073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.709064    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22df6623-5855-4e65-909a-669d291db280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38bf2752-kd8tt\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:16:57.810259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.810221    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22df6623-5855-4e65-909a-669d291db280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38bf2752-kd8tt\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:16:57.810455 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.810327    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n85z\" (UniqueName: \"kubernetes.io/projected/22df6623-5855-4e65-909a-669d291db280-kube-api-access-5n85z\") pod \"maas-api-revoke-sub-38bf2752-kd8tt\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:16:57.810455 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.810351    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22df6623-5855-4e65-909a-669d291db280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38bf2752-kd8tt\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:16:57.810954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.810936    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22df6623-5855-4e65-909a-669d291db280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38bf2752-kd8tt\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:16:57.812720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.812699    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22df6623-5855-4e65-909a-669d291db280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38bf2752-kd8tt\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:16:57.874365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.874296    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n85z\" (UniqueName: \"kubernetes.io/projected/22df6623-5855-4e65-909a-669d291db280-kube-api-access-5n85z\") pod \"maas-api-revoke-sub-38bf2752-kd8tt\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:16:57.909296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:57.909267    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:16:58.291034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:58.290999    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"]
Nov 03 07:16:59.168216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:59.167148    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt" event={"ID":"22df6623-5855-4e65-909a-669d291db280","Type":"ContainerStarted","Data":"af6b1667b978f22265cf5131c7df9ee370335e09f521faff2a3116c7ec9bd69d"}
Nov 03 07:16:59.168216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:16:59.167226    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt" event={"ID":"22df6623-5855-4e65-909a-669d291db280","Type":"ContainerStarted","Data":"2d6063bdce9a94ea2f0b04920bae755d438cdb19f3c2cc28737131fdff080746"}
Nov 03 07:17:00.174941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:00.174903    4470 generic.go:356] "Generic (PLEG): container finished" podID="22df6623-5855-4e65-909a-669d291db280" containerID="af6b1667b978f22265cf5131c7df9ee370335e09f521faff2a3116c7ec9bd69d" exitCode=0
Nov 03 07:17:00.175407 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:00.174995    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt" event={"ID":"22df6623-5855-4e65-909a-669d291db280","Type":"ContainerDied","Data":"af6b1667b978f22265cf5131c7df9ee370335e09f521faff2a3116c7ec9bd69d"}
Nov 03 07:17:00.284360 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:00.284306    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt" podStartSLOduration=3.284287767 podStartE2EDuration="3.284287767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:59.308258149 +0000 UTC m=+1071.640627138" watchObservedRunningTime="2026-11-03 07:17:00.284287767 +0000 UTC m=+1072.616656810"
Nov 03 07:17:00.286209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:00.286185    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"]
Nov 03 07:17:00.286411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:00.286386    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" podUID="155e6cf3-4920-4d89-b783-7125c6974485" containerName="istio-proxy" containerID="cri-o://b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21" gracePeriod=30
Nov 03 07:17:01.329482 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:01.329459    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:17:01.498532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:01.498446    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22df6623-5855-4e65-909a-669d291db280-kube-api-access-5n85z\" (UniqueName: \"kubernetes.io/projected/22df6623-5855-4e65-909a-669d291db280-kube-api-access-5n85z\") pod \"22df6623-5855-4e65-909a-669d291db280\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") "
Nov 03 07:17:01.498706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:01.498574    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22df6623-5855-4e65-909a-669d291db280-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22df6623-5855-4e65-909a-669d291db280-maas-api-xks-ca\") pod \"22df6623-5855-4e65-909a-669d291db280\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") "
Nov 03 07:17:01.498706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:01.498609    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22df6623-5855-4e65-909a-669d291db280-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22df6623-5855-4e65-909a-669d291db280-maas-api-openshift-ca\") pod \"22df6623-5855-4e65-909a-669d291db280\" (UID: \"22df6623-5855-4e65-909a-669d291db280\") "
Nov 03 07:17:01.499033 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:01.499001    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22df6623-5855-4e65-909a-669d291db280-maas-api-openshift-ca" pod "22df6623-5855-4e65-909a-669d291db280" (UID: "22df6623-5855-4e65-909a-669d291db280"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:17:01.500857 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:01.500833    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22df6623-5855-4e65-909a-669d291db280-kube-api-access-5n85z" pod "22df6623-5855-4e65-909a-669d291db280" (UID: "22df6623-5855-4e65-909a-669d291db280"). InnerVolumeSpecName "kube-api-access-5n85z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:17:01.500977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:01.500949    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22df6623-5855-4e65-909a-669d291db280-maas-api-xks-ca" pod "22df6623-5855-4e65-909a-669d291db280" (UID: "22df6623-5855-4e65-909a-669d291db280"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:17:01.600254 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:01.600223    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n85z\" (UniqueName: \"kubernetes.io/projected/22df6623-5855-4e65-909a-669d291db280-kube-api-access-5n85z\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:01.600254 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:01.600253    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22df6623-5855-4e65-909a-669d291db280-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:01.600442 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:01.600263    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22df6623-5855-4e65-909a-669d291db280-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:02.188462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:02.188391    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"
Nov 03 07:17:02.188462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:02.188398    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt" event={"ID":"22df6623-5855-4e65-909a-669d291db280","Type":"ContainerDied","Data":"2d6063bdce9a94ea2f0b04920bae755d438cdb19f3c2cc28737131fdff080746"}
Nov 03 07:17:02.188462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:02.188443    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d6063bdce9a94ea2f0b04920bae755d438cdb19f3c2cc28737131fdff080746"
Nov 03 07:17:04.759689 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:17:04.759655    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:17:05.542953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.542927    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:17:05.647039 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647005    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-envoy\") pod \"155e6cf3-4920-4d89-b783-7125c6974485\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") "
Nov 03 07:17:05.647243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647095    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-istio-token\" (UniqueName: \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-istio-token\") pod \"155e6cf3-4920-4d89-b783-7125c6974485\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") "
Nov 03 07:17:05.647243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647125    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/155e6cf3-4920-4d89-b783-7125c6974485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/155e6cf3-4920-4d89-b783-7125c6974485-istiod-ca-cert\") pod \"155e6cf3-4920-4d89-b783-7125c6974485\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") "
Nov 03 07:17:05.647243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647192    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/155e6cf3-4920-4d89-b783-7125c6974485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/155e6cf3-4920-4d89-b783-7125c6974485-istio-podinfo\") pod \"155e6cf3-4920-4d89-b783-7125c6974485\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") "
Nov 03 07:17:05.647425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647266    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-credential-socket\") pod \"155e6cf3-4920-4d89-b783-7125c6974485\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") "
Nov 03 07:17:05.647517 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647483    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-data\") pod \"155e6cf3-4920-4d89-b783-7125c6974485\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") "
Nov 03 07:17:05.647587 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647530    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-certs\") pod \"155e6cf3-4920-4d89-b783-7125c6974485\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") "
Nov 03 07:17:05.647587 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647581    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-socket\") pod \"155e6cf3-4920-4d89-b783-7125c6974485\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") "
Nov 03 07:17:05.647710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647576    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-credential-socket" pod "155e6cf3-4920-4d89-b783-7125c6974485" (UID: "155e6cf3-4920-4d89-b783-7125c6974485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:17:05.647710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647624    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-kube-api-access-x2p49\" (UniqueName: \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-kube-api-access-x2p49\") pod \"155e6cf3-4920-4d89-b783-7125c6974485\" (UID: \"155e6cf3-4920-4d89-b783-7125c6974485\") "
Nov 03 07:17:05.647813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647786    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-data" pod "155e6cf3-4920-4d89-b783-7125c6974485" (UID: "155e6cf3-4920-4d89-b783-7125c6974485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:17:05.647813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647799    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-certs" pod "155e6cf3-4920-4d89-b783-7125c6974485" (UID: "155e6cf3-4920-4d89-b783-7125c6974485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:17:05.647813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.647708    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155e6cf3-4920-4d89-b783-7125c6974485-istiod-ca-cert" pod "155e6cf3-4920-4d89-b783-7125c6974485" (UID: "155e6cf3-4920-4d89-b783-7125c6974485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:17:05.648253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.648229    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/155e6cf3-4920-4d89-b783-7125c6974485-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:05.648253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.648239    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-socket" pod "155e6cf3-4920-4d89-b783-7125c6974485" (UID: "155e6cf3-4920-4d89-b783-7125c6974485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:17:05.648253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.648254    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:05.648453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.648269    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:05.648453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.648282    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:05.649829 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.649803    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-istio-token" pod "155e6cf3-4920-4d89-b783-7125c6974485" (UID: "155e6cf3-4920-4d89-b783-7125c6974485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:17:05.649948 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.649833    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/155e6cf3-4920-4d89-b783-7125c6974485-istio-podinfo" pod "155e6cf3-4920-4d89-b783-7125c6974485" (UID: "155e6cf3-4920-4d89-b783-7125c6974485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:17:05.650118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.650096    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-kube-api-access-x2p49" pod "155e6cf3-4920-4d89-b783-7125c6974485" (UID: "155e6cf3-4920-4d89-b783-7125c6974485"). InnerVolumeSpecName "kube-api-access-x2p49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:17:05.650434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.650401    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-envoy" pod "155e6cf3-4920-4d89-b783-7125c6974485" (UID: "155e6cf3-4920-4d89-b783-7125c6974485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:17:05.749866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.749827    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:05.749866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.749858    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/155e6cf3-4920-4d89-b783-7125c6974485-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:05.750059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.749875    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:05.750059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.749886    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2p49\" (UniqueName: \"kubernetes.io/projected/155e6cf3-4920-4d89-b783-7125c6974485-kube-api-access-x2p49\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:05.750059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:05.749896    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/155e6cf3-4920-4d89-b783-7125c6974485-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:06.214591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:06.214560    4470 generic.go:356] "Generic (PLEG): container finished" podID="155e6cf3-4920-4d89-b783-7125c6974485" containerID="b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21" exitCode=0
Nov 03 07:17:06.214994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:06.214604    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"
Nov 03 07:17:06.214994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:06.214642    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" event={"ID":"155e6cf3-4920-4d89-b783-7125c6974485","Type":"ContainerDied","Data":"b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21"}
Nov 03 07:17:06.214994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:06.214699    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc" event={"ID":"155e6cf3-4920-4d89-b783-7125c6974485","Type":"ContainerDied","Data":"ebc4a6e8081a77d2800f94358240f96f5fb60ecbc7fadaa651938e2fa7471e80"}
Nov 03 07:17:06.214994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:06.214727    4470 scope.go:122] "RemoveContainer" containerID="b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21"
Nov 03 07:17:06.228456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:06.228063    4470 scope.go:122] "RemoveContainer" containerID="b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21"
Nov 03 07:17:06.228622 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:17:06.228604    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21\": container with ID starting with b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21 not found: ID does not exist" containerID="b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21"
Nov 03 07:17:06.228676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:06.228630    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21"} err="failed to get container status \"b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21\": rpc error: code = NotFound desc = could not find container \"b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21\": container with ID starting with b65570e90008948dc98d0a404f758857b0b46a5e23e42b26668a1946593f0c21 not found: ID does not exist"
Nov 03 07:17:06.405897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:06.405866    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"]
Nov 03 07:17:06.417005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:06.416980    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc"]
Nov 03 07:17:07.807723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:07.807690    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="155e6cf3-4920-4d89-b783-7125c6974485" path="/var/lib/kubelet/pods/155e6cf3-4920-4d89-b783-7125c6974485/volumes"
Nov 03 07:17:08.691982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.691946    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"]
Nov 03 07:17:08.692516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.692502    4470 cpu_manager.go:414] "containerMap: removing container" podUID="155e6cf3-4920-4d89-b783-7125c6974485" containerName="istio-proxy"
Nov 03 07:17:08.692580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.692518    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="155e6cf3-4920-4d89-b783-7125c6974485" containerName="istio-proxy"
Nov 03 07:17:08.692580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.692540    4470 cpu_manager.go:414] "containerMap: removing container" podUID="22df6623-5855-4e65-909a-669d291db280" containerName="revoke-keys"
Nov 03 07:17:08.692580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.692545    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22df6623-5855-4e65-909a-669d291db280" containerName="revoke-keys"
Nov 03 07:17:08.692671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.692616    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="22df6623-5855-4e65-909a-669d291db280" containerName="revoke-keys"
Nov 03 07:17:08.692671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.692630    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="155e6cf3-4920-4d89-b783-7125c6974485" containerName="istio-proxy"
Nov 03 07:17:08.697448 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.697431    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:08.755937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.755906    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"]
Nov 03 07:17:08.780475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.780431    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmgxd\" (UniqueName: \"kubernetes.io/projected/88cb1aab-be65-43e4-aa18-8063a399efb0-kube-api-access-cmgxd\") pod \"maas-api-revoke-sub-d37d5456-wnwqw\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:08.780475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.780478    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d37d5456-wnwqw\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:08.780644 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.780567    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d37d5456-wnwqw\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:08.802354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.802319    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:17:08.802354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.802352    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:17:08.802619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.802605    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:17:08.802726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.802713    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:17:08.881466 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.881434    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmgxd\" (UniqueName: \"kubernetes.io/projected/88cb1aab-be65-43e4-aa18-8063a399efb0-kube-api-access-cmgxd\") pod \"maas-api-revoke-sub-d37d5456-wnwqw\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:08.881922 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.881473    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d37d5456-wnwqw\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:08.881922 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.881497    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d37d5456-wnwqw\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:08.882184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.882125    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d37d5456-wnwqw\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:08.883951 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.883930    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d37d5456-wnwqw\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:08.946284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:08.946219    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmgxd\" (UniqueName: \"kubernetes.io/projected/88cb1aab-be65-43e4-aa18-8063a399efb0-kube-api-access-cmgxd\") pod \"maas-api-revoke-sub-d37d5456-wnwqw\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:09.010011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.009984    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:09.391269 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:17:09.391238    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88cb1aab_be65_43e4_aa18_8063a399efb0.slice/crio-c16b123f62d640837fd68afc008d8ac1ea72d1b862c9b52564474a7c11cc9b13 WatchSource:0}: Error finding container c16b123f62d640837fd68afc008d8ac1ea72d1b862c9b52564474a7c11cc9b13: Status 404 returned error can't find the container with id c16b123f62d640837fd68afc008d8ac1ea72d1b862c9b52564474a7c11cc9b13
Nov 03 07:17:09.391395 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.391317    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"]
Nov 03 07:17:09.577826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.577797    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"]
Nov 03 07:17:09.583067 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.583050    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:09.590042 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.590016    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6c7915-f95bb\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:09.590166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.590050    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4g6s\" (UniqueName: \"kubernetes.io/projected/127a1ab7-123f-4276-9657-475cbadb8389-kube-api-access-t4g6s\") pod \"maas-api-revoke-sub-0c6c7915-f95bb\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:09.590230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.590208    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6c7915-f95bb\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:09.642703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.642621    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"]
Nov 03 07:17:09.691620 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.691574    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6c7915-f95bb\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:09.691620 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.691624    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4g6s\" (UniqueName: \"kubernetes.io/projected/127a1ab7-123f-4276-9657-475cbadb8389-kube-api-access-t4g6s\") pod \"maas-api-revoke-sub-0c6c7915-f95bb\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:09.691821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.691667    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6c7915-f95bb\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:09.692296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.692266    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6c7915-f95bb\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:09.694022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.693997    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6c7915-f95bb\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:09.756501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.756471    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4g6s\" (UniqueName: \"kubernetes.io/projected/127a1ab7-123f-4276-9657-475cbadb8389-kube-api-access-t4g6s\") pod \"maas-api-revoke-sub-0c6c7915-f95bb\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:09.896400 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:09.896330    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:10.243045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:10.242957    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw" event={"ID":"88cb1aab-be65-43e4-aa18-8063a399efb0","Type":"ContainerStarted","Data":"cdf924c841627b06eb5539755aa98b4637186586873046e7f8a6ec16ee8afd24"}
Nov 03 07:17:10.243205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:10.243050    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw" event={"ID":"88cb1aab-be65-43e4-aa18-8063a399efb0","Type":"ContainerStarted","Data":"c16b123f62d640837fd68afc008d8ac1ea72d1b862c9b52564474a7c11cc9b13"}
Nov 03 07:17:10.277385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:10.277360    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"]
Nov 03 07:17:10.309502 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:17:10.309472    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127a1ab7_123f_4276_9657_475cbadb8389.slice/crio-ea449a6bfbd44b7b821106634e573cc9a01e62df403b60d76294a1cc6b425963 WatchSource:0}: Error finding container ea449a6bfbd44b7b821106634e573cc9a01e62df403b60d76294a1cc6b425963: Status 404 returned error can't find the container with id ea449a6bfbd44b7b821106634e573cc9a01e62df403b60d76294a1cc6b425963
Nov 03 07:17:10.363398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:10.363344    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw" podStartSLOduration=2.363329742 podStartE2EDuration="2.363329742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:10.362844097 +0000 UTC m=+1082.695213091" watchObservedRunningTime="2026-11-03 07:17:10.363329742 +0000 UTC m=+1082.695698770"
Nov 03 07:17:11.251191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:11.251162    4470 generic.go:356] "Generic (PLEG): container finished" podID="88cb1aab-be65-43e4-aa18-8063a399efb0" containerID="cdf924c841627b06eb5539755aa98b4637186586873046e7f8a6ec16ee8afd24" exitCode=0
Nov 03 07:17:11.251628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:11.251228    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw" event={"ID":"88cb1aab-be65-43e4-aa18-8063a399efb0","Type":"ContainerDied","Data":"cdf924c841627b06eb5539755aa98b4637186586873046e7f8a6ec16ee8afd24"}
Nov 03 07:17:11.252735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:11.252713    4470 generic.go:356] "Generic (PLEG): container finished" podID="127a1ab7-123f-4276-9657-475cbadb8389" containerID="972092f0dfc2cdf094b45a999ba4ed88d31ddee9da7cfdcdc165d041a03f5fb9" exitCode=0
Nov 03 07:17:11.252829 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:11.252787    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb" event={"ID":"127a1ab7-123f-4276-9657-475cbadb8389","Type":"ContainerDied","Data":"972092f0dfc2cdf094b45a999ba4ed88d31ddee9da7cfdcdc165d041a03f5fb9"}
Nov 03 07:17:11.252829 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:11.252825    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb" event={"ID":"127a1ab7-123f-4276-9657-475cbadb8389","Type":"ContainerStarted","Data":"ea449a6bfbd44b7b821106634e573cc9a01e62df403b60d76294a1cc6b425963"}
Nov 03 07:17:12.447826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.447802    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:12.450816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.450796    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:12.519750 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.519682    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-openshift-ca\") pod \"88cb1aab-be65-43e4-aa18-8063a399efb0\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") "
Nov 03 07:17:12.519750 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.519727    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-openshift-ca\") pod \"127a1ab7-123f-4276-9657-475cbadb8389\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") "
Nov 03 07:17:12.519947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.519856    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88cb1aab-be65-43e4-aa18-8063a399efb0-kube-api-access-cmgxd\" (UniqueName: \"kubernetes.io/projected/88cb1aab-be65-43e4-aa18-8063a399efb0-kube-api-access-cmgxd\") pod \"88cb1aab-be65-43e4-aa18-8063a399efb0\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") "
Nov 03 07:17:12.519947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.519874    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127a1ab7-123f-4276-9657-475cbadb8389-kube-api-access-t4g6s\" (UniqueName: \"kubernetes.io/projected/127a1ab7-123f-4276-9657-475cbadb8389-kube-api-access-t4g6s\") pod \"127a1ab7-123f-4276-9657-475cbadb8389\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") "
Nov 03 07:17:12.519947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.519913    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-xks-ca\") pod \"127a1ab7-123f-4276-9657-475cbadb8389\" (UID: \"127a1ab7-123f-4276-9657-475cbadb8389\") "
Nov 03 07:17:12.520185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.519971    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-xks-ca\") pod \"88cb1aab-be65-43e4-aa18-8063a399efb0\" (UID: \"88cb1aab-be65-43e4-aa18-8063a399efb0\") "
Nov 03 07:17:12.520185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.520053    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-openshift-ca" pod "88cb1aab-be65-43e4-aa18-8063a399efb0" (UID: "88cb1aab-be65-43e4-aa18-8063a399efb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:17:12.520185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.520057    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-openshift-ca" pod "127a1ab7-123f-4276-9657-475cbadb8389" (UID: "127a1ab7-123f-4276-9657-475cbadb8389"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:17:12.520518 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.520467    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:12.520518 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.520496    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:12.522036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.522006    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127a1ab7-123f-4276-9657-475cbadb8389-kube-api-access-t4g6s" pod "127a1ab7-123f-4276-9657-475cbadb8389" (UID: "127a1ab7-123f-4276-9657-475cbadb8389"). InnerVolumeSpecName "kube-api-access-t4g6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:17:12.522163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.522052    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-xks-ca" pod "127a1ab7-123f-4276-9657-475cbadb8389" (UID: "127a1ab7-123f-4276-9657-475cbadb8389"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:17:12.522163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.522099    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88cb1aab-be65-43e4-aa18-8063a399efb0-kube-api-access-cmgxd" pod "88cb1aab-be65-43e4-aa18-8063a399efb0" (UID: "88cb1aab-be65-43e4-aa18-8063a399efb0"). InnerVolumeSpecName "kube-api-access-cmgxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:17:12.522289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.522272    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-xks-ca" pod "88cb1aab-be65-43e4-aa18-8063a399efb0" (UID: "88cb1aab-be65-43e4-aa18-8063a399efb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:17:12.621598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.621569    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmgxd\" (UniqueName: \"kubernetes.io/projected/88cb1aab-be65-43e4-aa18-8063a399efb0-kube-api-access-cmgxd\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:12.621598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.621594    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4g6s\" (UniqueName: \"kubernetes.io/projected/127a1ab7-123f-4276-9657-475cbadb8389-kube-api-access-t4g6s\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:12.621598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.621605    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127a1ab7-123f-4276-9657-475cbadb8389-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:12.621836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:12.621614    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88cb1aab-be65-43e4-aa18-8063a399efb0-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:13.267981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:13.267957    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"
Nov 03 07:17:13.268200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:13.267954    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw" event={"ID":"88cb1aab-be65-43e4-aa18-8063a399efb0","Type":"ContainerDied","Data":"c16b123f62d640837fd68afc008d8ac1ea72d1b862c9b52564474a7c11cc9b13"}
Nov 03 07:17:13.268200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:13.268078    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c16b123f62d640837fd68afc008d8ac1ea72d1b862c9b52564474a7c11cc9b13"
Nov 03 07:17:13.269346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:13.269327    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"
Nov 03 07:17:13.269441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:13.269378    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb" event={"ID":"127a1ab7-123f-4276-9657-475cbadb8389","Type":"ContainerDied","Data":"ea449a6bfbd44b7b821106634e573cc9a01e62df403b60d76294a1cc6b425963"}
Nov 03 07:17:13.269441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:13.269417    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea449a6bfbd44b7b821106634e573cc9a01e62df403b60d76294a1cc6b425963"
Nov 03 07:17:22.050699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.050609    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"]
Nov 03 07:17:22.051693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.051417    4470 cpu_manager.go:414] "containerMap: removing container" podUID="88cb1aab-be65-43e4-aa18-8063a399efb0" containerName="revoke-keys"
Nov 03 07:17:22.051693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.051436    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88cb1aab-be65-43e4-aa18-8063a399efb0" containerName="revoke-keys"
Nov 03 07:17:22.051693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.051503    4470 cpu_manager.go:414] "containerMap: removing container" podUID="127a1ab7-123f-4276-9657-475cbadb8389" containerName="revoke-keys"
Nov 03 07:17:22.051693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.051511    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127a1ab7-123f-4276-9657-475cbadb8389" containerName="revoke-keys"
Nov 03 07:17:22.051693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.051618    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="88cb1aab-be65-43e4-aa18-8063a399efb0" containerName="revoke-keys"
Nov 03 07:17:22.051693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.051636    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="127a1ab7-123f-4276-9657-475cbadb8389" containerName="revoke-keys"
Nov 03 07:17:22.056864 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.056780    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.133219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.133185    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"]
Nov 03 07:17:22.153880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.153848    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.154055 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.153896    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed0fc15-6376-4298-a439-fb8f692b82a7-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.154055 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.154030    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.154184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.154060    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.154184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.154081    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.154342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.154317    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.154387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.154375    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.154433 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.154417    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.154568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.154552    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpsq2\" (UniqueName: \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-kube-api-access-wpsq2\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.180584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.180555    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3fea3a-openshift-default-dockercfg-9452b\""
Nov 03 07:17:22.256219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.256158    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpsq2\" (UniqueName: \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-kube-api-access-wpsq2\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.256409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.256232    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.256409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.256259    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed0fc15-6376-4298-a439-fb8f692b82a7-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.256409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.256312    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.256409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.256333    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.256409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.256352    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.256682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.256425    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.256682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.256456    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.256682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.256481    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.256886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.256858    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.256886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.256879    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.257082 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.257060    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.257225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.257204    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.257279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.257238    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed0fc15-6376-4298-a439-fb8f692b82a7-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.258856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.258833    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.258947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.258923    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.322476 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.322440    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.322476 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.322455    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpsq2\" (UniqueName: \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-kube-api-access-wpsq2\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.373960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.373926    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:22.806118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.806094    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"]
Nov 03 07:17:22.806488 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:17:22.806466    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed0fc15_6376_4298_a439_fb8f692b82a7.slice/crio-01499c59268a33d0db5e16dae68367e6ff430561c168d4cf52b6759c94c304ae WatchSource:0}: Error finding container 01499c59268a33d0db5e16dae68367e6ff430561c168d4cf52b6759c94c304ae: Status 404 returned error can't find the container with id 01499c59268a33d0db5e16dae68367e6ff430561c168d4cf52b6759c94c304ae
Nov 03 07:17:22.808710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.808677    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:17:22.808773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.808744    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:17:22.808810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:22.808774    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:17:23.345249 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:23.345217    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" event={"ID":"4ed0fc15-6376-4298-a439-fb8f692b82a7","Type":"ContainerStarted","Data":"63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4"}
Nov 03 07:17:23.345654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:23.345270    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" event={"ID":"4ed0fc15-6376-4298-a439-fb8f692b82a7","Type":"ContainerStarted","Data":"01499c59268a33d0db5e16dae68367e6ff430561c168d4cf52b6759c94c304ae"}
Nov 03 07:17:23.374207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:23.374164    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:23.375990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:23.375964    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 07:17:23.376175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:23.376038    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" podUID="4ed0fc15-6376-4298-a439-fb8f692b82a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 07:17:23.477471 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:23.475102    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" podStartSLOduration=1.475087353 podStartE2EDuration="1.475087353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:23.470669812 +0000 UTC m=+1095.803038807" watchObservedRunningTime="2026-11-03 07:17:23.475087353 +0000 UTC m=+1095.807456407"
Nov 03 07:17:24.375220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:24.375183    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 07:17:24.375647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:24.375286    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" podUID="4ed0fc15-6376-4298-a439-fb8f692b82a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 07:17:25.374413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:25.374378    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 07:17:25.374640 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:25.374440    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" podUID="4ed0fc15-6376-4298-a439-fb8f692b82a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 07:17:26.374776 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:26.374741    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 07:17:26.377346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:26.374802    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" podUID="4ed0fc15-6376-4298-a439-fb8f692b82a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 07:17:27.374876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:27.374836    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 07:17:27.375350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:27.374895    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" podUID="4ed0fc15-6376-4298-a439-fb8f692b82a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 07:17:28.374864 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:28.374830    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 07:17:28.375049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:28.374894    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" podUID="4ed0fc15-6376-4298-a439-fb8f692b82a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 07:17:29.465792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:29.465762    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:29.466449 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:29.466426    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:29.467304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:29.467285    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:17:31.416961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.416935    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"]
Nov 03 07:17:31.421107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.421091    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:31.480088 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.480059    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"]
Nov 03 07:17:31.526460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.526428    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:17:31.526654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.526622    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:17:31.526703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.526692    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:17:31.526746 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.526727    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:17:31.580808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.580764    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5876a62a-nmg9n\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:31.580986 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.580818    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nhm5\" (UniqueName: \"kubernetes.io/projected/c22d4004-93ee-459c-a354-cce7ac13680e-kube-api-access-5nhm5\") pod \"maas-api-revoke-sub-5876a62a-nmg9n\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:31.581100 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.581085    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5876a62a-nmg9n\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:31.682577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.682488    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5876a62a-nmg9n\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:31.682577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.682528    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nhm5\" (UniqueName: \"kubernetes.io/projected/c22d4004-93ee-459c-a354-cce7ac13680e-kube-api-access-5nhm5\") pod \"maas-api-revoke-sub-5876a62a-nmg9n\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:31.682808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.682649    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5876a62a-nmg9n\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:31.683243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.683214    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5876a62a-nmg9n\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:31.685014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.684995    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5876a62a-nmg9n\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:31.747112 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:31.747080    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nhm5\" (UniqueName: \"kubernetes.io/projected/c22d4004-93ee-459c-a354-cce7ac13680e-kube-api-access-5nhm5\") pod \"maas-api-revoke-sub-5876a62a-nmg9n\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:32.035281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:32.035202    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:32.414940 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:17:32.414911    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22d4004_93ee_459c_a354_cce7ac13680e.slice/crio-e6b56d906ce432da2a63671612aa29a2dcc1539f6534ef701777729d1dda9a99 WatchSource:0}: Error finding container e6b56d906ce432da2a63671612aa29a2dcc1539f6534ef701777729d1dda9a99: Status 404 returned error can't find the container with id e6b56d906ce432da2a63671612aa29a2dcc1539f6534ef701777729d1dda9a99
Nov 03 07:17:32.415064 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:32.415006    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"]
Nov 03 07:17:32.426628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:32.426597    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n" event={"ID":"c22d4004-93ee-459c-a354-cce7ac13680e","Type":"ContainerStarted","Data":"e6b56d906ce432da2a63671612aa29a2dcc1539f6534ef701777729d1dda9a99"}
Nov 03 07:17:33.435944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:33.435904    4470 generic.go:356] "Generic (PLEG): container finished" podID="c22d4004-93ee-459c-a354-cce7ac13680e" containerID="70dd99aa8848dde63457ac364cea9609666c6d14170d5904d532bb6594feb122" exitCode=0
Nov 03 07:17:33.436369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:33.435994    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n" event={"ID":"c22d4004-93ee-459c-a354-cce7ac13680e","Type":"ContainerDied","Data":"70dd99aa8848dde63457ac364cea9609666c6d14170d5904d532bb6594feb122"}
Nov 03 07:17:34.597474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:34.597451    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:34.720404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:34.720299    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-openshift-ca\") pod \"c22d4004-93ee-459c-a354-cce7ac13680e\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") "
Nov 03 07:17:34.720573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:34.720431    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-xks-ca\") pod \"c22d4004-93ee-459c-a354-cce7ac13680e\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") "
Nov 03 07:17:34.720573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:34.720494    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22d4004-93ee-459c-a354-cce7ac13680e-kube-api-access-5nhm5\" (UniqueName: \"kubernetes.io/projected/c22d4004-93ee-459c-a354-cce7ac13680e-kube-api-access-5nhm5\") pod \"c22d4004-93ee-459c-a354-cce7ac13680e\" (UID: \"c22d4004-93ee-459c-a354-cce7ac13680e\") "
Nov 03 07:17:34.720727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:34.720701    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-openshift-ca" pod "c22d4004-93ee-459c-a354-cce7ac13680e" (UID: "c22d4004-93ee-459c-a354-cce7ac13680e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:17:34.720990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:34.720966    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:34.722801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:34.722771    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-xks-ca" pod "c22d4004-93ee-459c-a354-cce7ac13680e" (UID: "c22d4004-93ee-459c-a354-cce7ac13680e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:17:34.722801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:34.722784    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22d4004-93ee-459c-a354-cce7ac13680e-kube-api-access-5nhm5" pod "c22d4004-93ee-459c-a354-cce7ac13680e" (UID: "c22d4004-93ee-459c-a354-cce7ac13680e"). InnerVolumeSpecName "kube-api-access-5nhm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:17:34.761169 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:17:34.761102    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:17:34.822285 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:34.822256    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22d4004-93ee-459c-a354-cce7ac13680e-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:34.822285 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:34.822284    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nhm5\" (UniqueName: \"kubernetes.io/projected/c22d4004-93ee-459c-a354-cce7ac13680e-kube-api-access-5nhm5\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:35.450959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:35.450920    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n" event={"ID":"c22d4004-93ee-459c-a354-cce7ac13680e","Type":"ContainerDied","Data":"e6b56d906ce432da2a63671612aa29a2dcc1539f6534ef701777729d1dda9a99"}
Nov 03 07:17:35.450959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:35.450942    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"
Nov 03 07:17:35.451232 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:35.450969    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6b56d906ce432da2a63671612aa29a2dcc1539f6534ef701777729d1dda9a99"
Nov 03 07:17:39.481089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:39.481048    4470 generic.go:356] "Generic (PLEG): container finished" podID="913409c5-9471-495c-8556-92596081537d" containerID="2f41ab02aabc5c2a05f0e878258e687082389f6d1481151fe01ca20c968f278a" exitCode=0
Nov 03 07:17:39.481089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:39.481085    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9hf2w-pxwm6" event={"ID":"913409c5-9471-495c-8556-92596081537d","Type":"ContainerDied","Data":"2f41ab02aabc5c2a05f0e878258e687082389f6d1481151fe01ca20c968f278a"}
Nov 03 07:17:40.224559 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.224524    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"]
Nov 03 07:17:40.225042 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.225030    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c22d4004-93ee-459c-a354-cce7ac13680e" containerName="revoke-keys"
Nov 03 07:17:40.225086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.225044    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22d4004-93ee-459c-a354-cce7ac13680e" containerName="revoke-keys"
Nov 03 07:17:40.225181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.225171    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22d4004-93ee-459c-a354-cce7ac13680e" containerName="revoke-keys"
Nov 03 07:17:40.228825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.228805    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.284156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.284110    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-dshm\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.284341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.284245    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-home\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.284341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.284299    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-model-cache\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.284341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.284338    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-tmp-dir\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.284505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.284460    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-strr5\" (UniqueName: \"kubernetes.io/projected/bf127fad-1026-4c13-ad33-28a806e7ebca-kube-api-access-strr5\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.284558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.284544    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf127fad-1026-4c13-ad33-28a806e7ebca-tls-certs\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.334509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.334477    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3fea3a\"/\"distinct-w5-3fea3a-kserve-self-signed-certs\""
Nov 03 07:17:40.338072 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.338049    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"]
Nov 03 07:17:40.385523 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.385490    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-dshm\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.385675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.385563    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-home\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.385675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.385614    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-model-cache\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.385675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.385644    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-tmp-dir\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.385675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.385664    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-strr5\" (UniqueName: \"kubernetes.io/projected/bf127fad-1026-4c13-ad33-28a806e7ebca-kube-api-access-strr5\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.385894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.385733    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf127fad-1026-4c13-ad33-28a806e7ebca-tls-certs\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.386053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.386030    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-home\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.386191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.386065    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-model-cache\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.386191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.386119    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-tmp-dir\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.388281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.388260    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-dshm\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.389567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.389542    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf127fad-1026-4c13-ad33-28a806e7ebca-tls-certs\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.450718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.450691    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-strr5\" (UniqueName: \"kubernetes.io/projected/bf127fad-1026-4c13-ad33-28a806e7ebca-kube-api-access-strr5\") pod \"distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.541172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.541126    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:40.674316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.674291    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:17:40.806414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.806321    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913409c5-9471-495c-8556-92596081537d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/913409c5-9471-495c-8556-92596081537d-archives-path\") pod \"913409c5-9471-495c-8556-92596081537d\" (UID: \"913409c5-9471-495c-8556-92596081537d\") "
Nov 03 07:17:40.806414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.806376    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913409c5-9471-495c-8556-92596081537d-kube-api-access-blqr7\" (UniqueName: \"kubernetes.io/projected/913409c5-9471-495c-8556-92596081537d-kube-api-access-blqr7\") pod \"913409c5-9471-495c-8556-92596081537d\" (UID: \"913409c5-9471-495c-8556-92596081537d\") "
Nov 03 07:17:40.806414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.806413    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-service-ca-bundle\") pod \"913409c5-9471-495c-8556-92596081537d\" (UID: \"913409c5-9471-495c-8556-92596081537d\") "
Nov 03 07:17:40.806703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.806524    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-trusted-ca-bundle\") pod \"913409c5-9471-495c-8556-92596081537d\" (UID: \"913409c5-9471-495c-8556-92596081537d\") "
Nov 03 07:17:40.806764 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.806696    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913409c5-9471-495c-8556-92596081537d-archives-path" pod "913409c5-9471-495c-8556-92596081537d" (UID: "913409c5-9471-495c-8556-92596081537d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:17:40.806857 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.806839    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-service-ca-bundle" pod "913409c5-9471-495c-8556-92596081537d" (UID: "913409c5-9471-495c-8556-92596081537d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:17:40.806999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.806973    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-trusted-ca-bundle" pod "913409c5-9471-495c-8556-92596081537d" (UID: "913409c5-9471-495c-8556-92596081537d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:17:40.807073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.807059    4470 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-trusted-ca-bundle\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:40.807161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.807075    4470 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/913409c5-9471-495c-8556-92596081537d-archives-path\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:40.807161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.807085    4470 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913409c5-9471-495c-8556-92596081537d-service-ca-bundle\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:40.808778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.808753    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913409c5-9471-495c-8556-92596081537d-kube-api-access-blqr7" pod "913409c5-9471-495c-8556-92596081537d" (UID: "913409c5-9471-495c-8556-92596081537d"). InnerVolumeSpecName "kube-api-access-blqr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:17:40.908167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.908118    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blqr7\" (UniqueName: \"kubernetes.io/projected/913409c5-9471-495c-8556-92596081537d-kube-api-access-blqr7\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:17:40.965425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:40.965395    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"]
Nov 03 07:17:40.965738 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:17:40.965712    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf127fad_1026_4c13_ad33_28a806e7ebca.slice/crio-1da32daddb2a7ad9373c7b128c182b1d5acc704f03c3df0e5f914158ed05aad8 WatchSource:0}: Error finding container 1da32daddb2a7ad9373c7b128c182b1d5acc704f03c3df0e5f914158ed05aad8: Status 404 returned error can't find the container with id 1da32daddb2a7ad9373c7b128c182b1d5acc704f03c3df0e5f914158ed05aad8
Nov 03 07:17:41.496657 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:41.496620    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" event={"ID":"bf127fad-1026-4c13-ad33-28a806e7ebca","Type":"ContainerStarted","Data":"b3c342f84e1e0b9059ebbbd75f2e6251fa4250bae1a2a56d12c5b3cc5d48676f"}
Nov 03 07:17:41.496820 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:41.496666    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" event={"ID":"bf127fad-1026-4c13-ad33-28a806e7ebca","Type":"ContainerStarted","Data":"1da32daddb2a7ad9373c7b128c182b1d5acc704f03c3df0e5f914158ed05aad8"}
Nov 03 07:17:41.498093 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:41.498073    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9hf2w-pxwm6"
Nov 03 07:17:41.498226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:41.498108    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9hf2w-pxwm6" event={"ID":"913409c5-9471-495c-8556-92596081537d","Type":"ContainerDied","Data":"2272645670ac239a08e2e92ea2eaf0c66f4082fc3cfd177bc8931b95530b7202"}
Nov 03 07:17:41.498226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:41.498187    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2272645670ac239a08e2e92ea2eaf0c66f4082fc3cfd177bc8931b95530b7202"
Nov 03 07:17:41.541651 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:41.541626    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:41.621408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:41.621358    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podStartSLOduration=1.621344633 podStartE2EDuration="1.621344633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:41.617589769 +0000 UTC m=+1113.949958762" watchObservedRunningTime="2026-11-03 07:17:41.621344633 +0000 UTC m=+1113.953713640"
Nov 03 07:17:50.541981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:50.541895    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:50.554999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:50.554975    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:50.572919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:50.572895    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:17:58.053229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.053191    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"]
Nov 03 07:17:58.053740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.053727    4470 cpu_manager.go:414] "containerMap: removing container" podUID="913409c5-9471-495c-8556-92596081537d" containerName="insights-gathering"
Nov 03 07:17:58.053783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.053741    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913409c5-9471-495c-8556-92596081537d" containerName="insights-gathering"
Nov 03 07:17:58.053844 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.053836    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="913409c5-9471-495c-8556-92596081537d" containerName="insights-gathering"
Nov 03 07:17:58.057277 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.057256    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.121304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.121265    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"]
Nov 03 07:17:58.165592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.165560    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"distinct-w2-669d6b-kserve-self-signed-certs\""
Nov 03 07:17:58.218191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.216307    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tls-certs\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.218191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.216406    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-home\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.218191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.216474    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-dshm\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.218191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.216497    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tmp-dir\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.218191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.216937    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-model-cache\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.218191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.217015    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhj6\" (UniqueName: \"kubernetes.io/projected/6e10c86c-6b75-4686-937b-55c11fbf9ae9-kube-api-access-dwhj6\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.317890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.317853    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tls-certs\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.318080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.317916    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-home\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.318080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.317962    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-dshm\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.318080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.317986    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tmp-dir\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.318288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.318090    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-model-cache\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.318288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.318183    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhj6\" (UniqueName: \"kubernetes.io/projected/6e10c86c-6b75-4686-937b-55c11fbf9ae9-kube-api-access-dwhj6\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.318506 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.318482    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-home\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.318586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.318503    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-model-cache\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.318586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.318556    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tmp-dir\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.320503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.320485    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-dshm\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.320759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.320743    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tls-certs\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.382802 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.382773    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhj6\" (UniqueName: \"kubernetes.io/projected/6e10c86c-6b75-4686-937b-55c11fbf9ae9-kube-api-access-dwhj6\") pod \"distinct-w2-669d6b-kserve-6f78f6b654-62lvp\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:58.669882 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:58.669800    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:59.050165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:59.050124    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"]
Nov 03 07:17:59.051019 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:17:59.050996    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e10c86c_6b75_4686_937b_55c11fbf9ae9.slice/crio-77f141f08c6da7adfbf33e67743ff9d42526e7c3b5324836588b7971aa939066 WatchSource:0}: Error finding container 77f141f08c6da7adfbf33e67743ff9d42526e7c3b5324836588b7971aa939066: Status 404 returned error can't find the container with id 77f141f08c6da7adfbf33e67743ff9d42526e7c3b5324836588b7971aa939066
Nov 03 07:17:59.624038 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:59.624005    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" event={"ID":"6e10c86c-6b75-4686-937b-55c11fbf9ae9","Type":"ContainerStarted","Data":"dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350"}
Nov 03 07:17:59.624457 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:59.624046    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" event={"ID":"6e10c86c-6b75-4686-937b-55c11fbf9ae9","Type":"ContainerStarted","Data":"77f141f08c6da7adfbf33e67743ff9d42526e7c3b5324836588b7971aa939066"}
Nov 03 07:17:59.670877 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:59.670852    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:17:59.746575 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:17:59.746524    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podStartSLOduration=1.746512306 podStartE2EDuration="1.746512306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:59.744970156 +0000 UTC m=+1132.077339150" watchObservedRunningTime="2026-11-03 07:17:59.746512306 +0000 UTC m=+1132.078881301"
Nov 03 07:18:04.761807 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:18:04.761768    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:18:08.670492 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:08.670458    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:18:08.684253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:08.684225    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:18:08.696573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:08.696547    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:18:34.761596 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:18:34.761553    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:18:38.171120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.171086    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"]
Nov 03 07:18:38.175357 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.175340    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.241501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.241469    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"]
Nov 03 07:18:38.271847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.271816    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"kube-root-ca.crt\""
Nov 03 07:18:38.271847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.271821    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"openshift-service-ca.crt\""
Nov 03 07:18:38.281626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.281600    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"default-dockercfg-tkq4z\""
Nov 03 07:18:38.281626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.281619    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"distinct-2-w3-1f2e7e-kserve-self-signed-certs\""
Nov 03 07:18:38.289253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.289225    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68plg\" (UniqueName: \"kubernetes.io/projected/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-kube-api-access-68plg\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.289389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.289269    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tls-certs\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.289389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.289332    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tmp-dir\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.289640 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.289624    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-home\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.289754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.289738    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-model-cache\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.289802 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.289767    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-dshm\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.391103 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.391068    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-home\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.391335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.391202    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-model-cache\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.391335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.391232    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-dshm\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.391335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.391272    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68plg\" (UniqueName: \"kubernetes.io/projected/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-kube-api-access-68plg\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.391335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.391304    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tls-certs\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.391595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.391353    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tmp-dir\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.391595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.391558    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-home\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.391686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.391626    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-model-cache\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.391766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.391744    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tmp-dir\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.393711 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.393690    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-dshm\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.393961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.393942    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tls-certs\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.456269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.456200    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68plg\" (UniqueName: \"kubernetes.io/projected/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-kube-api-access-68plg\") pod \"distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.487100 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.487055    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:38.886329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:38.886300    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"]
Nov 03 07:18:38.886630 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:18:38.886609    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60fa78f7_82a6_4265_bafc_3f09f8cb7d39.slice/crio-6ccd8d732fc5c0745a3509ec37b51f1a56506de7b696175e1b5f24c32f03c041 WatchSource:0}: Error finding container 6ccd8d732fc5c0745a3509ec37b51f1a56506de7b696175e1b5f24c32f03c041: Status 404 returned error can't find the container with id 6ccd8d732fc5c0745a3509ec37b51f1a56506de7b696175e1b5f24c32f03c041
Nov 03 07:18:39.891377 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:39.891341    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" event={"ID":"60fa78f7-82a6-4265-bafc-3f09f8cb7d39","Type":"ContainerStarted","Data":"f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0"}
Nov 03 07:18:39.891377 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:39.891386    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" event={"ID":"60fa78f7-82a6-4265-bafc-3f09f8cb7d39","Type":"ContainerStarted","Data":"6ccd8d732fc5c0745a3509ec37b51f1a56506de7b696175e1b5f24c32f03c041"}
Nov 03 07:18:40.017945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:40.017893    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podStartSLOduration=2.017879728 podStartE2EDuration="2.017879728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:18:40.014398946 +0000 UTC m=+1172.346767940" watchObservedRunningTime="2026-11-03 07:18:40.017879728 +0000 UTC m=+1172.350248723"
Nov 03 07:18:40.487427 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:40.487396    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:48.487872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:48.487838    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:48.501502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:48.501480    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:18:48.963941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:18:48.963912    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:19:04.761109 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:19:04.761068    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:19:19.618076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.617998    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"]
Nov 03 07:19:19.621976 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.621960    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.683326 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.683289    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"]
Nov 03 07:19:19.687557 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.687512    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3fea3a\"/\"distinct-2-w5-3fea3a-kserve-self-signed-certs\""
Nov 03 07:19:19.789502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.789474    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-tmp-dir\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.789732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.789516    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-model-cache\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.789732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.789671    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-home\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.789851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.789783    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4tpb\" (UniqueName: \"kubernetes.io/projected/62350718-e181-41ff-83b1-4d05a66539a7-kube-api-access-x4tpb\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.789851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.789830    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-dshm\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.789954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.789870    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62350718-e181-41ff-83b1-4d05a66539a7-tls-certs\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.891308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.891224    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4tpb\" (UniqueName: \"kubernetes.io/projected/62350718-e181-41ff-83b1-4d05a66539a7-kube-api-access-x4tpb\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.891308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.891290    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-dshm\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.891550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.891343    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62350718-e181-41ff-83b1-4d05a66539a7-tls-certs\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.891550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.891424    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-tmp-dir\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.891550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.891462    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-model-cache\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.891550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.891536    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-home\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.891888 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.891857    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-tmp-dir\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.892011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.891936    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-model-cache\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.892011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.891950    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-home\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.894291 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.894262    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-dshm\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.894578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.894560    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62350718-e181-41ff-83b1-4d05a66539a7-tls-certs\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:19.959568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:19.959529    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4tpb\" (UniqueName: \"kubernetes.io/projected/62350718-e181-41ff-83b1-4d05a66539a7-kube-api-access-x4tpb\") pod \"distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") " pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:20.234317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:20.234232    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:20.618086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:20.618058    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"]
Nov 03 07:19:20.618501 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:19:20.618479    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62350718_e181_41ff_83b1_4d05a66539a7.slice/crio-81fb170e026f0f8b05690ac67728bc04286fdb20379cff2b27c632e39d8c8677 WatchSource:0}: Error finding container 81fb170e026f0f8b05690ac67728bc04286fdb20379cff2b27c632e39d8c8677: Status 404 returned error can't find the container with id 81fb170e026f0f8b05690ac67728bc04286fdb20379cff2b27c632e39d8c8677
Nov 03 07:19:21.189064 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:21.189031    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" event={"ID":"62350718-e181-41ff-83b1-4d05a66539a7","Type":"ContainerStarted","Data":"c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50"}
Nov 03 07:19:21.189295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:21.189077    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" event={"ID":"62350718-e181-41ff-83b1-4d05a66539a7","Type":"ContainerStarted","Data":"81fb170e026f0f8b05690ac67728bc04286fdb20379cff2b27c632e39d8c8677"}
Nov 03 07:19:21.234828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:21.234793    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:21.313884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:21.313834    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podStartSLOduration=2.313820992 podStartE2EDuration="2.313820992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:19:21.310350388 +0000 UTC m=+1213.642719393" watchObservedRunningTime="2026-11-03 07:19:21.313820992 +0000 UTC m=+1213.646189987"
Nov 03 07:19:30.235114 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:30.235076    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:30.248049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:30.248020    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:30.262884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:30.262862    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:19:34.760747 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:19:34.760712    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:19:47.882649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:47.882614    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"]
Nov 03 07:19:47.886973 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:47.886954    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:47.946075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:47.946048    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"]
Nov 03 07:19:47.992239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:47.992204    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:19:47.992415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:47.992245    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:19:47.992415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:47.992262    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:19:47.992415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:47.992339    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:19:48.011733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:48.011698    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7ce3ae9-t8drs\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:48.011902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:48.011884    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hsnd\" (UniqueName: \"kubernetes.io/projected/af623390-0e79-46d3-902e-15248baf6f6a-kube-api-access-6hsnd\") pod \"maas-api-revoke-sub-e7ce3ae9-t8drs\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:48.011949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:48.011920    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7ce3ae9-t8drs\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:48.113233 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:48.113197    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hsnd\" (UniqueName: \"kubernetes.io/projected/af623390-0e79-46d3-902e-15248baf6f6a-kube-api-access-6hsnd\") pod \"maas-api-revoke-sub-e7ce3ae9-t8drs\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:48.113436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:48.113241    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7ce3ae9-t8drs\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:48.113436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:48.113300    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7ce3ae9-t8drs\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:48.114096 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:48.114072    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7ce3ae9-t8drs\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:48.115804 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:48.115784    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7ce3ae9-t8drs\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:48.177813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:48.177739    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hsnd\" (UniqueName: \"kubernetes.io/projected/af623390-0e79-46d3-902e-15248baf6f6a-kube-api-access-6hsnd\") pod \"maas-api-revoke-sub-e7ce3ae9-t8drs\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:48.204361 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:48.204332    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:48.592915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:48.592885    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"]
Nov 03 07:19:48.593122 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:19:48.593105    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf623390_0e79_46d3_902e_15248baf6f6a.slice/crio-0b9f1b67103af0db00bc0ca79029a2233940abe9e2dabcbf1d35ee55539e8313 WatchSource:0}: Error finding container 0b9f1b67103af0db00bc0ca79029a2233940abe9e2dabcbf1d35ee55539e8313: Status 404 returned error can't find the container with id 0b9f1b67103af0db00bc0ca79029a2233940abe9e2dabcbf1d35ee55539e8313
Nov 03 07:19:49.373777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:49.373739    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs" event={"ID":"af623390-0e79-46d3-902e-15248baf6f6a","Type":"ContainerStarted","Data":"3abaa68d2a9bf401deec139d33d5c91fad9dc81dc62f44d02b530c4d384a3e8d"}
Nov 03 07:19:49.374220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:49.373803    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs" event={"ID":"af623390-0e79-46d3-902e-15248baf6f6a","Type":"ContainerStarted","Data":"0b9f1b67103af0db00bc0ca79029a2233940abe9e2dabcbf1d35ee55539e8313"}
Nov 03 07:19:49.496636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:49.496584    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs" podStartSLOduration=2.496570562 podStartE2EDuration="2.496570562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:19:49.493295393 +0000 UTC m=+1241.825664388" watchObservedRunningTime="2026-11-03 07:19:49.496570562 +0000 UTC m=+1241.828939569"
Nov 03 07:19:50.382459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:50.382424    4470 generic.go:356] "Generic (PLEG): container finished" podID="af623390-0e79-46d3-902e-15248baf6f6a" containerID="3abaa68d2a9bf401deec139d33d5c91fad9dc81dc62f44d02b530c4d384a3e8d" exitCode=0
Nov 03 07:19:50.382893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:50.382509    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs" event={"ID":"af623390-0e79-46d3-902e-15248baf6f6a","Type":"ContainerDied","Data":"3abaa68d2a9bf401deec139d33d5c91fad9dc81dc62f44d02b530c4d384a3e8d"}
Nov 03 07:19:51.542942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:51.542918    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:51.662378 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:51.662306    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-xks-ca\") pod \"af623390-0e79-46d3-902e-15248baf6f6a\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") "
Nov 03 07:19:51.662519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:51.662400    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-openshift-ca\") pod \"af623390-0e79-46d3-902e-15248baf6f6a\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") "
Nov 03 07:19:51.662519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:51.662433    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af623390-0e79-46d3-902e-15248baf6f6a-kube-api-access-6hsnd\" (UniqueName: \"kubernetes.io/projected/af623390-0e79-46d3-902e-15248baf6f6a-kube-api-access-6hsnd\") pod \"af623390-0e79-46d3-902e-15248baf6f6a\" (UID: \"af623390-0e79-46d3-902e-15248baf6f6a\") "
Nov 03 07:19:51.662731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:51.662711    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-openshift-ca" pod "af623390-0e79-46d3-902e-15248baf6f6a" (UID: "af623390-0e79-46d3-902e-15248baf6f6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:19:51.662928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:51.662906    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:19:51.664769 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:51.664743    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af623390-0e79-46d3-902e-15248baf6f6a-kube-api-access-6hsnd" pod "af623390-0e79-46d3-902e-15248baf6f6a" (UID: "af623390-0e79-46d3-902e-15248baf6f6a"). InnerVolumeSpecName "kube-api-access-6hsnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:19:51.664953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:51.664925    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-xks-ca" pod "af623390-0e79-46d3-902e-15248baf6f6a" (UID: "af623390-0e79-46d3-902e-15248baf6f6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:19:51.764265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:51.764229    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hsnd\" (UniqueName: \"kubernetes.io/projected/af623390-0e79-46d3-902e-15248baf6f6a-kube-api-access-6hsnd\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:19:51.764265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:51.764256    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af623390-0e79-46d3-902e-15248baf6f6a-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:19:52.397498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:52.397462    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs" event={"ID":"af623390-0e79-46d3-902e-15248baf6f6a","Type":"ContainerDied","Data":"0b9f1b67103af0db00bc0ca79029a2233940abe9e2dabcbf1d35ee55539e8313"}
Nov 03 07:19:52.397498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:52.397491    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"
Nov 03 07:19:52.397724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:52.397509    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9f1b67103af0db00bc0ca79029a2233940abe9e2dabcbf1d35ee55539e8313"
Nov 03 07:19:58.657958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.657924    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"]
Nov 03 07:19:58.658779 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.658762    4470 cpu_manager.go:414] "containerMap: removing container" podUID="af623390-0e79-46d3-902e-15248baf6f6a" containerName="revoke-keys"
Nov 03 07:19:58.658825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.658781    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af623390-0e79-46d3-902e-15248baf6f6a" containerName="revoke-keys"
Nov 03 07:19:58.658932 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.658921    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="af623390-0e79-46d3-902e-15248baf6f6a" containerName="revoke-keys"
Nov 03 07:19:58.662563 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.662546    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:19:58.720873 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.720842    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"]
Nov 03 07:19:58.742294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.742263    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb1952a2-zphwx\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:19:58.742454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.742342    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzj7b\" (UniqueName: \"kubernetes.io/projected/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-kube-api-access-tzj7b\") pod \"maas-api-revoke-sub-eb1952a2-zphwx\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:19:58.742500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.742467    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb1952a2-zphwx\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:19:58.766053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.766019    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:19:58.766298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.766115    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:19:58.767275 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.767253    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:19:58.767448 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.767362    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:19:58.844240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.844206    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb1952a2-zphwx\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:19:58.844418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.844272    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzj7b\" (UniqueName: \"kubernetes.io/projected/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-kube-api-access-tzj7b\") pod \"maas-api-revoke-sub-eb1952a2-zphwx\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:19:58.844418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.844364    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb1952a2-zphwx\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:19:58.845015 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.844983    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb1952a2-zphwx\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:19:58.846841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.846810    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb1952a2-zphwx\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:19:58.910518 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.910447    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzj7b\" (UniqueName: \"kubernetes.io/projected/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-kube-api-access-tzj7b\") pod \"maas-api-revoke-sub-eb1952a2-zphwx\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:19:58.976673 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:58.976638    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:19:59.376862 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:19:59.376834    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2582dd_7fc3_4c6a_aa5c_7e79388880e6.slice/crio-d2a279b42f47253e987af7809a252ab139ab42d220fd3a34dee82a189a9fd032 WatchSource:0}: Error finding container d2a279b42f47253e987af7809a252ab139ab42d220fd3a34dee82a189a9fd032: Status 404 returned error can't find the container with id d2a279b42f47253e987af7809a252ab139ab42d220fd3a34dee82a189a9fd032
Nov 03 07:19:59.377024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:59.376999    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"]
Nov 03 07:19:59.447056 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:19:59.447024    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx" event={"ID":"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6","Type":"ContainerStarted","Data":"d2a279b42f47253e987af7809a252ab139ab42d220fd3a34dee82a189a9fd032"}
Nov 03 07:20:00.456030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:00.455995    4470 generic.go:356] "Generic (PLEG): container finished" podID="2a2582dd-7fc3-4c6a-aa5c-7e79388880e6" containerID="1350e6e23cbe9d1bd5a05b8a090f2f17fd919e8645b603770a39a276593e24cf" exitCode=0
Nov 03 07:20:00.456448 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:00.456088    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx" event={"ID":"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6","Type":"ContainerDied","Data":"1350e6e23cbe9d1bd5a05b8a090f2f17fd919e8645b603770a39a276593e24cf"}
Nov 03 07:20:01.625652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:01.625624    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:20:01.693967 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:01.693880    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-kube-api-access-tzj7b\" (UniqueName: \"kubernetes.io/projected/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-kube-api-access-tzj7b\") pod \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") "
Nov 03 07:20:01.693967 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:01.693922    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-xks-ca\") pod \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") "
Nov 03 07:20:01.694242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:01.694025    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-openshift-ca\") pod \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\" (UID: \"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6\") "
Nov 03 07:20:01.694469 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:01.694444    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-openshift-ca" pod "2a2582dd-7fc3-4c6a-aa5c-7e79388880e6" (UID: "2a2582dd-7fc3-4c6a-aa5c-7e79388880e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:20:01.694558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:01.694538    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:01.696291 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:01.696270    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-kube-api-access-tzj7b" pod "2a2582dd-7fc3-4c6a-aa5c-7e79388880e6" (UID: "2a2582dd-7fc3-4c6a-aa5c-7e79388880e6"). InnerVolumeSpecName "kube-api-access-tzj7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:20:01.696364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:01.696331    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-xks-ca" pod "2a2582dd-7fc3-4c6a-aa5c-7e79388880e6" (UID: "2a2582dd-7fc3-4c6a-aa5c-7e79388880e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:20:01.796189 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:01.796157    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzj7b\" (UniqueName: \"kubernetes.io/projected/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-kube-api-access-tzj7b\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:01.796189 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:01.796187    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:02.471816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:02.471790    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"
Nov 03 07:20:02.471998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:02.471786    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx" event={"ID":"2a2582dd-7fc3-4c6a-aa5c-7e79388880e6","Type":"ContainerDied","Data":"d2a279b42f47253e987af7809a252ab139ab42d220fd3a34dee82a189a9fd032"}
Nov 03 07:20:02.471998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:02.471908    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2a279b42f47253e987af7809a252ab139ab42d220fd3a34dee82a189a9fd032"
Nov 03 07:20:04.079407 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:04.079370    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"]
Nov 03 07:20:04.129934 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:04.129891    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5fe16c8-5kffr"]
Nov 03 07:20:04.760198 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:20:04.760116    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:20:05.807791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:05.807755    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3fd299-28ea-4746-b665-95dd3c823ebe" path="/var/lib/kubelet/pods/ae3fd299-28ea-4746-b665-95dd3c823ebe/volumes"
Nov 03 07:20:06.295602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.295574    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"]
Nov 03 07:20:06.296168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.296145    4470 cpu_manager.go:414] "containerMap: removing container" podUID="2a2582dd-7fc3-4c6a-aa5c-7e79388880e6" containerName="revoke-keys"
Nov 03 07:20:06.296168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.296162    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2582dd-7fc3-4c6a-aa5c-7e79388880e6" containerName="revoke-keys"
Nov 03 07:20:06.296332 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.296257    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2582dd-7fc3-4c6a-aa5c-7e79388880e6" containerName="revoke-keys"
Nov 03 07:20:06.300183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.300164    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.367893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.367864    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"]
Nov 03 07:20:06.405893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.405863    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ffee69\"/\"distinct-2-w4-ffee69-kserve-self-signed-certs\""
Nov 03 07:20:06.451568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.451529    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-home\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.451777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.451584    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc62r\" (UniqueName: \"kubernetes.io/projected/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-kube-api-access-qc62r\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.451777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.451627    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tmp-dir\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.451777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.451743    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-model-cache\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.452162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.452095    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tls-certs\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.452390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.452368    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-dshm\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.553981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.553894    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-dshm\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.554125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.554011    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-home\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.554125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.554035    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc62r\" (UniqueName: \"kubernetes.io/projected/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-kube-api-access-qc62r\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.554125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.554059    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tmp-dir\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.554125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.554087    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-model-cache\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.554125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.554115    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tls-certs\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.554470 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.554451    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-home\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.554534 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.554508    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tmp-dir\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.554594 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.554536    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-model-cache\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.556385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.556358    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-dshm\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.556702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.556686    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tls-certs\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.619017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.618986    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc62r\" (UniqueName: \"kubernetes.io/projected/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-kube-api-access-qc62r\") pod \"distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:06.912393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:06.912357    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:07.309993 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:20:07.309965    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1ba71f7_2c52_49e3_845d_6f89f1d9157a.slice/crio-07ff8fc2fcb60909c641e9f9efe4cb62c01ff6018f3e6cd2632e10a32ed96d55 WatchSource:0}: Error finding container 07ff8fc2fcb60909c641e9f9efe4cb62c01ff6018f3e6cd2632e10a32ed96d55: Status 404 returned error can't find the container with id 07ff8fc2fcb60909c641e9f9efe4cb62c01ff6018f3e6cd2632e10a32ed96d55
Nov 03 07:20:07.311162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:07.311118    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"]
Nov 03 07:20:07.509526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:07.509482    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" event={"ID":"a1ba71f7-2c52-49e3-845d-6f89f1d9157a","Type":"ContainerStarted","Data":"6631380b5bc3110c484187abc55abfcd2aac8c32eb4f58199eaff72d2a22d603"}
Nov 03 07:20:07.509711 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:07.509542    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" event={"ID":"a1ba71f7-2c52-49e3-845d-6f89f1d9157a","Type":"ContainerStarted","Data":"07ff8fc2fcb60909c641e9f9efe4cb62c01ff6018f3e6cd2632e10a32ed96d55"}
Nov 03 07:20:07.636325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:07.636271    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podStartSLOduration=1.636256988 podStartE2EDuration="1.636256988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:20:07.633967942 +0000 UTC m=+1259.966336937" watchObservedRunningTime="2026-11-03 07:20:07.636256988 +0000 UTC m=+1259.968625982"
Nov 03 07:20:07.913353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:07.913198    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:16.060701 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.060667    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"]
Nov 03 07:20:16.111037 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.111006    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d853984-bpz2f"]
Nov 03 07:20:16.823697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.823663    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"]
Nov 03 07:20:16.883356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.878054    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"]
Nov 03 07:20:16.884928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.884896    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:16.912663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.912636    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:16.925556 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.925532    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:16.939943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.939902    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"]
Nov 03 07:20:16.941613 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.941595    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:20:16.942005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.941988    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:20:16.942708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.942690    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:20:16.985767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.985737    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:20:16.985925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.985836    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxdq\" (UniqueName: \"kubernetes.io/projected/793930bc-31b3-4c8d-9c4d-c272abfb8d96-kube-api-access-zdxdq\") pod \"maas-api-revoke-sub-7e636b1f-2vnpq\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:16.985973 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.985959    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e636b1f-2vnpq\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:16.986009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:16.985981    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e636b1f-2vnpq\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:17.087456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:17.087368    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxdq\" (UniqueName: \"kubernetes.io/projected/793930bc-31b3-4c8d-9c4d-c272abfb8d96-kube-api-access-zdxdq\") pod \"maas-api-revoke-sub-7e636b1f-2vnpq\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:17.087456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:17.087456    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e636b1f-2vnpq\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:17.087957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:17.087617    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e636b1f-2vnpq\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:17.088355 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:17.088338    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e636b1f-2vnpq\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:17.089833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:17.089811    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e636b1f-2vnpq\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:17.152168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:17.152102    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxdq\" (UniqueName: \"kubernetes.io/projected/793930bc-31b3-4c8d-9c4d-c272abfb8d96-kube-api-access-zdxdq\") pod \"maas-api-revoke-sub-7e636b1f-2vnpq\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:17.201505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:17.201476    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:17.578313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:17.578254    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerName="istio-proxy" containerID="cri-o://7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324" gracePeriod=30
Nov 03 07:20:17.586461 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:20:17.586424    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793930bc_31b3_4c8d_9c4d_c272abfb8d96.slice/crio-0cbd55d19a297226ae6c8cce3d8cfab4922673946708fd26bb240b8de31dd9dc WatchSource:0}: Error finding container 0cbd55d19a297226ae6c8cce3d8cfab4922673946708fd26bb240b8de31dd9dc: Status 404 returned error can't find the container with id 0cbd55d19a297226ae6c8cce3d8cfab4922673946708fd26bb240b8de31dd9dc
Nov 03 07:20:17.587553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:17.587522    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"]
Nov 03 07:20:17.593753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:17.593733    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:20:17.807954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:17.807920    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95072d50-f6f0-431a-b7e5-395a7c3dc2d7" path="/var/lib/kubelet/pods/95072d50-f6f0-431a-b7e5-395a7c3dc2d7/volumes"
Nov 03 07:20:18.587283 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:18.587250    4470 generic.go:356] "Generic (PLEG): container finished" podID="793930bc-31b3-4c8d-9c4d-c272abfb8d96" containerID="903dd8b9b1989fbce5602683cea32547325794e314de453aded763b1ffbc12e1" exitCode=0
Nov 03 07:20:18.587744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:18.587336    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq" event={"ID":"793930bc-31b3-4c8d-9c4d-c272abfb8d96","Type":"ContainerDied","Data":"903dd8b9b1989fbce5602683cea32547325794e314de453aded763b1ffbc12e1"}
Nov 03 07:20:18.587744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:18.587388    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq" event={"ID":"793930bc-31b3-4c8d-9c4d-c272abfb8d96","Type":"ContainerStarted","Data":"0cbd55d19a297226ae6c8cce3d8cfab4922673946708fd26bb240b8de31dd9dc"}
Nov 03 07:20:19.744546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:19.744524    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:19.819707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:19.819675    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-xks-ca\") pod \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") "
Nov 03 07:20:19.819883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:19.819783    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793930bc-31b3-4c8d-9c4d-c272abfb8d96-kube-api-access-zdxdq\" (UniqueName: \"kubernetes.io/projected/793930bc-31b3-4c8d-9c4d-c272abfb8d96-kube-api-access-zdxdq\") pod \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") "
Nov 03 07:20:19.819980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:19.819966    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-openshift-ca\") pod \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\" (UID: \"793930bc-31b3-4c8d-9c4d-c272abfb8d96\") "
Nov 03 07:20:19.820359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:19.820333    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-openshift-ca" pod "793930bc-31b3-4c8d-9c4d-c272abfb8d96" (UID: "793930bc-31b3-4c8d-9c4d-c272abfb8d96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:20:19.821781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:19.821764    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-xks-ca" pod "793930bc-31b3-4c8d-9c4d-c272abfb8d96" (UID: "793930bc-31b3-4c8d-9c4d-c272abfb8d96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:20:19.821942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:19.821924    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793930bc-31b3-4c8d-9c4d-c272abfb8d96-kube-api-access-zdxdq" pod "793930bc-31b3-4c8d-9c4d-c272abfb8d96" (UID: "793930bc-31b3-4c8d-9c4d-c272abfb8d96"). InnerVolumeSpecName "kube-api-access-zdxdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:20:19.922046 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:19.921968    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:19.922046 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:19.921995    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793930bc-31b3-4c8d-9c4d-c272abfb8d96-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:19.922046 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:19.922004    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxdq\" (UniqueName: \"kubernetes.io/projected/793930bc-31b3-4c8d-9c4d-c272abfb8d96-kube-api-access-zdxdq\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:20.601627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:20.601598    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"
Nov 03 07:20:20.601833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:20.601636    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq" event={"ID":"793930bc-31b3-4c8d-9c4d-c272abfb8d96","Type":"ContainerDied","Data":"0cbd55d19a297226ae6c8cce3d8cfab4922673946708fd26bb240b8de31dd9dc"}
Nov 03 07:20:20.601833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:20.601705    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cbd55d19a297226ae6c8cce3d8cfab4922673946708fd26bb240b8de31dd9dc"
Nov 03 07:20:21.296883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:21.296841    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:20:21.297335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:21.296906    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:20:22.541148 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.541103    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"]
Nov 03 07:20:22.541693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.541678    4470 cpu_manager.go:414] "containerMap: removing container" podUID="793930bc-31b3-4c8d-9c4d-c272abfb8d96" containerName="revoke-keys"
Nov 03 07:20:22.541747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.541695    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793930bc-31b3-4c8d-9c4d-c272abfb8d96" containerName="revoke-keys"
Nov 03 07:20:22.541822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.541813    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="793930bc-31b3-4c8d-9c4d-c272abfb8d96" containerName="revoke-keys"
Nov 03 07:20:22.545525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.545510    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.637819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.637787    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"]
Nov 03 07:20:22.657051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.657016    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.657257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.657164    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a39c230-3238-4a0d-89c4-073b1ba5a429-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.657257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.657220    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89rs\" (UniqueName: \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-kube-api-access-v89rs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.657348 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.657277    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.657408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.657383    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.657536 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.657521    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.657581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.657556    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.657614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.657589    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.657659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.657620    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.758825    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.758921    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a39c230-3238-4a0d-89c4-073b1ba5a429-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.758951    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89rs\" (UniqueName: \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-kube-api-access-v89rs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.758973    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.759015    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.759086    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.759109    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.759168    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.759201    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.759559    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.760046    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.760591    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a39c230-3238-4a0d-89c4-073b1ba5a429-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.761069    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.761301    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.764325    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.768368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.764825    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.824789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.824762    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.824936 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.824918    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89rs\" (UniqueName: \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-kube-api-access-v89rs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.841159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.841115    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:20:22.859123 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.859105    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:22.960276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.960246    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-token\" (UniqueName: \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-token\") pod \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") "
Nov 03 07:20:22.960437 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.960383    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-kube-api-access-hj8qm\" (UniqueName: \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-kube-api-access-hj8qm\") pod \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") "
Nov 03 07:20:22.960484 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.960441    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-data\") pod \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") "
Nov 03 07:20:22.960544 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.960529    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-credential-socket\") pod \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") "
Nov 03 07:20:22.960614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.960600    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-certs\") pod \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") "
Nov 03 07:20:22.960689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.960674    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istiod-ca-cert\") pod \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") "
Nov 03 07:20:22.960761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.960722    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-socket\") pod \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") "
Nov 03 07:20:22.960761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.960756    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-envoy\") pod \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") "
Nov 03 07:20:22.960867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.960807    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-podinfo\") pod \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\" (UID: \"881fb74b-0ac3-4299-a577-852f5e6e6ed3\") "
Nov 03 07:20:22.960867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.960845    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-data" pod "881fb74b-0ac3-4299-a577-852f5e6e6ed3" (UID: "881fb74b-0ac3-4299-a577-852f5e6e6ed3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:20:22.960991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.960945    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-certs" pod "881fb74b-0ac3-4299-a577-852f5e6e6ed3" (UID: "881fb74b-0ac3-4299-a577-852f5e6e6ed3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:20:22.961157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.961111    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-credential-socket" pod "881fb74b-0ac3-4299-a577-852f5e6e6ed3" (UID: "881fb74b-0ac3-4299-a577-852f5e6e6ed3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:20:22.961244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.961224    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-socket" pod "881fb74b-0ac3-4299-a577-852f5e6e6ed3" (UID: "881fb74b-0ac3-4299-a577-852f5e6e6ed3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:20:22.961548 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.961517    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istiod-ca-cert" pod "881fb74b-0ac3-4299-a577-852f5e6e6ed3" (UID: "881fb74b-0ac3-4299-a577-852f5e6e6ed3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:20:22.961548 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.961524    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:22.961884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.961559    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:22.961884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.961576    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:22.961884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.961592    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:22.963043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.963019    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-token" pod "881fb74b-0ac3-4299-a577-852f5e6e6ed3" (UID: "881fb74b-0ac3-4299-a577-852f5e6e6ed3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:20:22.964081 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.964060    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-podinfo" pod "881fb74b-0ac3-4299-a577-852f5e6e6ed3" (UID: "881fb74b-0ac3-4299-a577-852f5e6e6ed3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:20:22.964492 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.964468    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-kube-api-access-hj8qm" pod "881fb74b-0ac3-4299-a577-852f5e6e6ed3" (UID: "881fb74b-0ac3-4299-a577-852f5e6e6ed3"). InnerVolumeSpecName "kube-api-access-hj8qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:20:22.965747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:22.965728    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-envoy" pod "881fb74b-0ac3-4299-a577-852f5e6e6ed3" (UID: "881fb74b-0ac3-4299-a577-852f5e6e6ed3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:20:23.063076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.063054    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj8qm\" (UniqueName: \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-kube-api-access-hj8qm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:23.063247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.063081    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:23.063247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.063095    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:23.063247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.063106    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:23.063247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.063114    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/881fb74b-0ac3-4299-a577-852f5e6e6ed3-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:23.295906 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:20:23.295874    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a39c230_3238_4a0d_89c4_073b1ba5a429.slice/crio-192817ed497e0e43810e12cef01fbd111be6e02efedaa9291b6800a37824a66b WatchSource:0}: Error finding container 192817ed497e0e43810e12cef01fbd111be6e02efedaa9291b6800a37824a66b: Status 404 returned error can't find the container with id 192817ed497e0e43810e12cef01fbd111be6e02efedaa9291b6800a37824a66b
Nov 03 07:20:23.297044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.297015    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"]
Nov 03 07:20:23.298718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.298690    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:20:23.298792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.298748    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:20:23.298792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.298776    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:20:23.622285 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.622244    4470 generic.go:356] "Generic (PLEG): container finished" podID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerID="7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324" exitCode=0
Nov 03 07:20:23.622729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.622314    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"
Nov 03 07:20:23.622729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.622331    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" event={"ID":"881fb74b-0ac3-4299-a577-852f5e6e6ed3","Type":"ContainerDied","Data":"7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324"}
Nov 03 07:20:23.622729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.622437    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69" event={"ID":"881fb74b-0ac3-4299-a577-852f5e6e6ed3","Type":"ContainerDied","Data":"06a2fdbb3da3506ae37d9696fe3ef5b5a54618ca615d5702d52e47cb9e2b41b8"}
Nov 03 07:20:23.622729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.622470    4470 scope.go:122] "RemoveContainer" containerID="7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324"
Nov 03 07:20:23.624755 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.624727    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" event={"ID":"8a39c230-3238-4a0d-89c4-073b1ba5a429","Type":"ContainerStarted","Data":"dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77"}
Nov 03 07:20:23.624887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.624779    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" event={"ID":"8a39c230-3238-4a0d-89c4-073b1ba5a429","Type":"ContainerStarted","Data":"192817ed497e0e43810e12cef01fbd111be6e02efedaa9291b6800a37824a66b"}
Nov 03 07:20:23.633165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.633123    4470 scope.go:122] "RemoveContainer" containerID="7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324"
Nov 03 07:20:23.633442 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:20:23.633423    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324\": container with ID starting with 7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324 not found: ID does not exist" containerID="7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324"
Nov 03 07:20:23.633532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.633449    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324"} err="failed to get container status \"7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324\": rpc error: code = NotFound desc = could not find container \"7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324\": container with ID starting with 7b9e4f628f4f75d9da8356f2f515115254ee2ebbfe075065291d914db9e09324 not found: ID does not exist"
Nov 03 07:20:23.805650 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.805603    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" podStartSLOduration=1.8055892629999999 podStartE2EDuration="1.805589263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:20:23.802836319 +0000 UTC m=+1276.135205316" watchObservedRunningTime="2026-11-03 07:20:23.805589263 +0000 UTC m=+1276.137958256"
Nov 03 07:20:23.859783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.859704    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:23.922246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.922214    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"]
Nov 03 07:20:23.929421 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:23.929398    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69"]
Nov 03 07:20:24.872501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:24.872471    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 07:20:24.872908 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:24.872539    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 07:20:25.806191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:25.806121    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" path="/var/lib/kubelet/pods/881fb74b-0ac3-4299-a577-852f5e6e6ed3/volumes"
Nov 03 07:20:25.859632 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:25.859571    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 07:20:25.859995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:25.859945    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 07:20:26.864614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:26.864575    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 07:20:26.865275 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:26.865240    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 07:20:27.870159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:27.866173    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 07:20:27.870159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:27.866251    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 07:20:28.871278 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:28.871241    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 07:20:28.871808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:28.871292    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 07:20:29.859826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:29.859791    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 07:20:29.860018 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:29.859847    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 07:20:31.071540 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:31.071507    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"]
Nov 03 07:20:31.129973 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:31.129925    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1320b63a-fdgv8"]
Nov 03 07:20:31.821021 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:31.820988    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4971a2c-a168-41fb-8daa-ec350882a57f" path="/var/lib/kubelet/pods/e4971a2c-a168-41fb-8daa-ec350882a57f/volumes"
Nov 03 07:20:31.861584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:31.861540    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:20:31.861787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:31.861623    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:20:31.908716 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:31.908687    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:31.909191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:31.909165    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:31.910076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:31.910056    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:20:32.590868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.590842    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"]
Nov 03 07:20:32.591585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.591564    4470 cpu_manager.go:414] "containerMap: removing container" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerName="istio-proxy"
Nov 03 07:20:32.591706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.591586    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerName="istio-proxy"
Nov 03 07:20:32.591815 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.591801    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="881fb74b-0ac3-4299-a577-852f5e6e6ed3" containerName="istio-proxy"
Nov 03 07:20:32.596304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.596284    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:32.620950 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.620926    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nc6h\" (UniqueName: \"kubernetes.io/projected/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-kube-api-access-4nc6h\") pod \"maas-api-revoke-sub-ead06e14-8rhlj\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:32.621059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.621044    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ead06e14-8rhlj\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:32.621172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.621154    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ead06e14-8rhlj\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:32.655385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.655362    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:20:32.655539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.655522    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:20:32.655643 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.655628    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:20:32.656992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.656974    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"]
Nov 03 07:20:32.701935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.701908    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:20:32.722602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.722573    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ead06e14-8rhlj\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:32.722746 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.722627    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ead06e14-8rhlj\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:32.722746 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.722701    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nc6h\" (UniqueName: \"kubernetes.io/projected/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-kube-api-access-4nc6h\") pod \"maas-api-revoke-sub-ead06e14-8rhlj\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:32.723249 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.723226    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ead06e14-8rhlj\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:32.725057 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.725037    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ead06e14-8rhlj\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:32.791246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.791217    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nc6h\" (UniqueName: \"kubernetes.io/projected/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-kube-api-access-4nc6h\") pod \"maas-api-revoke-sub-ead06e14-8rhlj\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:32.909169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:32.909066    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:33.301701 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:20:33.301667    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43babcf9_6a25_49e8_a37b_5b4ced2e6ba0.slice/crio-aa60eaa4cc83650c4cd4fb97fcbc83bf6a231f69bc71e45813ff5323abdbb95a WatchSource:0}: Error finding container aa60eaa4cc83650c4cd4fb97fcbc83bf6a231f69bc71e45813ff5323abdbb95a: Status 404 returned error can't find the container with id aa60eaa4cc83650c4cd4fb97fcbc83bf6a231f69bc71e45813ff5323abdbb95a
Nov 03 07:20:33.302181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:33.302154    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"]
Nov 03 07:20:33.727120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:33.727080    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj" event={"ID":"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0","Type":"ContainerStarted","Data":"bcdf9441ffe3784476739d7db5e1d4b7a36f9b6778215a626f21f1f37b972460"}
Nov 03 07:20:33.727546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:33.727173    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj" event={"ID":"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0","Type":"ContainerStarted","Data":"aa60eaa4cc83650c4cd4fb97fcbc83bf6a231f69bc71e45813ff5323abdbb95a"}
Nov 03 07:20:33.853686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:33.853636    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj" podStartSLOduration=1.853623699 podStartE2EDuration="1.853623699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:20:33.8528597 +0000 UTC m=+1286.185228694" watchObservedRunningTime="2026-11-03 07:20:33.853623699 +0000 UTC m=+1286.185992692"
Nov 03 07:20:34.735580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:34.735546    4470 generic.go:356] "Generic (PLEG): container finished" podID="43babcf9-6a25-49e8-a37b-5b4ced2e6ba0" containerID="bcdf9441ffe3784476739d7db5e1d4b7a36f9b6778215a626f21f1f37b972460" exitCode=0
Nov 03 07:20:34.735965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:34.735642    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj" event={"ID":"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0","Type":"ContainerDied","Data":"bcdf9441ffe3784476739d7db5e1d4b7a36f9b6778215a626f21f1f37b972460"}
Nov 03 07:20:34.762537 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:20:34.762507    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:20:35.899771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:35.899745    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:35.982422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:35.982390    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-kube-api-access-4nc6h\" (UniqueName: \"kubernetes.io/projected/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-kube-api-access-4nc6h\") pod \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") "
Nov 03 07:20:35.982601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:35.982434    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-xks-ca\") pod \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") "
Nov 03 07:20:35.982601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:35.982542    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-openshift-ca\") pod \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\" (UID: \"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0\") "
Nov 03 07:20:35.982866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:35.982841    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-openshift-ca" pod "43babcf9-6a25-49e8-a37b-5b4ced2e6ba0" (UID: "43babcf9-6a25-49e8-a37b-5b4ced2e6ba0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:20:35.983091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:35.983073    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:35.984818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:35.984790    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-xks-ca" pod "43babcf9-6a25-49e8-a37b-5b4ced2e6ba0" (UID: "43babcf9-6a25-49e8-a37b-5b4ced2e6ba0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:20:35.984958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:35.984937    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-kube-api-access-4nc6h" pod "43babcf9-6a25-49e8-a37b-5b4ced2e6ba0" (UID: "43babcf9-6a25-49e8-a37b-5b4ced2e6ba0"). InnerVolumeSpecName "kube-api-access-4nc6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:20:36.084009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:36.083978    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nc6h\" (UniqueName: \"kubernetes.io/projected/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-kube-api-access-4nc6h\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:36.084009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:36.084008    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:36.753512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:36.753466    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj" event={"ID":"43babcf9-6a25-49e8-a37b-5b4ced2e6ba0","Type":"ContainerDied","Data":"aa60eaa4cc83650c4cd4fb97fcbc83bf6a231f69bc71e45813ff5323abdbb95a"}
Nov 03 07:20:36.753512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:36.753494    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"
Nov 03 07:20:36.753730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:36.753549    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa60eaa4cc83650c4cd4fb97fcbc83bf6a231f69bc71e45813ff5323abdbb95a"
Nov 03 07:20:44.722658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.722624    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"]
Nov 03 07:20:44.723198 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.723184    4470 cpu_manager.go:414] "containerMap: removing container" podUID="43babcf9-6a25-49e8-a37b-5b4ced2e6ba0" containerName="revoke-keys"
Nov 03 07:20:44.723246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.723199    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43babcf9-6a25-49e8-a37b-5b4ced2e6ba0" containerName="revoke-keys"
Nov 03 07:20:44.723308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.723299    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="43babcf9-6a25-49e8-a37b-5b4ced2e6ba0" containerName="revoke-keys"
Nov 03 07:20:44.726981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.726966    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.787657 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.787625    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"]
Nov 03 07:20:44.833157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.833107    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6606d6\"/\"kube-root-ca.crt\""
Nov 03 07:20:44.833326 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.833309    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6606d6\"/\"openshift-service-ca.crt\""
Nov 03 07:20:44.833422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.833405    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6606d6\"/\"default-dockercfg-wk9lj\""
Nov 03 07:20:44.833461 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.833406    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6606d6\"/\"distinct-2-w0-6606d6-kserve-self-signed-certs\""
Nov 03 07:20:44.886433 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.886405    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-tmp-dir\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.886604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.886450    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-home\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.886604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.886584    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-dshm\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.886691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.886648    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmvg\" (UniqueName: \"kubernetes.io/projected/4d2a3693-ff83-4324-ba34-5722fdb42861-kube-api-access-5zmvg\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.886790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.886768    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-model-cache\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.886828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.886798    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2a3693-ff83-4324-ba34-5722fdb42861-tls-certs\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.987805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.987722    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmvg\" (UniqueName: \"kubernetes.io/projected/4d2a3693-ff83-4324-ba34-5722fdb42861-kube-api-access-5zmvg\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.987953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.987817    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-model-cache\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.987953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.987842    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2a3693-ff83-4324-ba34-5722fdb42861-tls-certs\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.987953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.987864    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-tmp-dir\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.987953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.987901    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-home\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.988127 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.988007    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-dshm\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.988370 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.988345    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-model-cache\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.988447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.988375    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-tmp-dir\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.988503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.988474    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-home\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.990468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.990441    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-dshm\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:44.990635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:44.990618    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2a3693-ff83-4324-ba34-5722fdb42861-tls-certs\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:45.051944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:45.051912    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmvg\" (UniqueName: \"kubernetes.io/projected/4d2a3693-ff83-4324-ba34-5722fdb42861-kube-api-access-5zmvg\") pod \"distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:45.341106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:45.341078    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:45.725108 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:20:45.725083    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d2a3693_ff83_4324_ba34_5722fdb42861.slice/crio-a730c6f9ccd5a1406e448b2e48c91d536143a2cf7681baff0dca43f26eb1a397 WatchSource:0}: Error finding container a730c6f9ccd5a1406e448b2e48c91d536143a2cf7681baff0dca43f26eb1a397: Status 404 returned error can't find the container with id a730c6f9ccd5a1406e448b2e48c91d536143a2cf7681baff0dca43f26eb1a397
Nov 03 07:20:45.725664 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:45.725641    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"]
Nov 03 07:20:45.819780 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:45.819746    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" event={"ID":"4d2a3693-ff83-4324-ba34-5722fdb42861","Type":"ContainerStarted","Data":"651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa"}
Nov 03 07:20:45.819910 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:45.819787    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" event={"ID":"4d2a3693-ff83-4324-ba34-5722fdb42861","Type":"ContainerStarted","Data":"a730c6f9ccd5a1406e448b2e48c91d536143a2cf7681baff0dca43f26eb1a397"}
Nov 03 07:20:45.943303 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:45.943255    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podStartSLOduration=1.943239835 podStartE2EDuration="1.943239835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:20:45.940407389 +0000 UTC m=+1298.272776388" watchObservedRunningTime="2026-11-03 07:20:45.943239835 +0000 UTC m=+1298.275608859"
Nov 03 07:20:46.341787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:46.341749    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:50.136215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.136119    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"]
Nov 03 07:20:50.140504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.140479    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:50.200505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.200471    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"]
Nov 03 07:20:50.245454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.245426    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:20:50.245454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.245427    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:20:50.245681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.245437    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:20:50.245681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.245578    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:20:50.248841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.248819    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa215abf-l4tqn\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:50.248913 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.248884    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jnkg\" (UniqueName: \"kubernetes.io/projected/362d8b24-79e2-4289-8e58-71942688b8ea-kube-api-access-9jnkg\") pod \"maas-api-revoke-sub-aa215abf-l4tqn\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:50.248982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.248964    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa215abf-l4tqn\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:50.350576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.350548    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa215abf-l4tqn\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:50.350770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.350655    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jnkg\" (UniqueName: \"kubernetes.io/projected/362d8b24-79e2-4289-8e58-71942688b8ea-kube-api-access-9jnkg\") pod \"maas-api-revoke-sub-aa215abf-l4tqn\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:50.350770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.350703    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa215abf-l4tqn\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:50.351357 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.351328    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa215abf-l4tqn\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:50.353221 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.353201    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa215abf-l4tqn\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:50.416176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.416095    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jnkg\" (UniqueName: \"kubernetes.io/projected/362d8b24-79e2-4289-8e58-71942688b8ea-kube-api-access-9jnkg\") pod \"maas-api-revoke-sub-aa215abf-l4tqn\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:50.454289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.454261    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:50.838192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.838159    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"]
Nov 03 07:20:50.838757 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:20:50.838730    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362d8b24_79e2_4289_8e58_71942688b8ea.slice/crio-08a3fcf0f82a39472fddf447f834eb3281c3605631fe23c79a084bc119650226 WatchSource:0}: Error finding container 08a3fcf0f82a39472fddf447f834eb3281c3605631fe23c79a084bc119650226: Status 404 returned error can't find the container with id 08a3fcf0f82a39472fddf447f834eb3281c3605631fe23c79a084bc119650226
Nov 03 07:20:50.855735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:50.855702    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn" event={"ID":"362d8b24-79e2-4289-8e58-71942688b8ea","Type":"ContainerStarted","Data":"08a3fcf0f82a39472fddf447f834eb3281c3605631fe23c79a084bc119650226"}
Nov 03 07:20:51.064281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:51.064254    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"]
Nov 03 07:20:51.072572 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:51.072546    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8c1943-wfprl"]
Nov 03 07:20:51.806701 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:51.806667    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae16c5f0-15ee-4899-83a0-41b3bb8de8d4" path="/var/lib/kubelet/pods/ae16c5f0-15ee-4899-83a0-41b3bb8de8d4/volumes"
Nov 03 07:20:51.864237 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:51.864198    4470 generic.go:356] "Generic (PLEG): container finished" podID="362d8b24-79e2-4289-8e58-71942688b8ea" containerID="1c5e74e74e7a69bc36a98ebddbf6fb8189d7801fb4a45965618c567a280adb5f" exitCode=0
Nov 03 07:20:51.864405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:51.864277    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn" event={"ID":"362d8b24-79e2-4289-8e58-71942688b8ea","Type":"ContainerDied","Data":"1c5e74e74e7a69bc36a98ebddbf6fb8189d7801fb4a45965618c567a280adb5f"}
Nov 03 07:20:53.036697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.036670    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:53.181105 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.181026    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-xks-ca\") pod \"362d8b24-79e2-4289-8e58-71942688b8ea\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") "
Nov 03 07:20:53.181276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.181234    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-openshift-ca\") pod \"362d8b24-79e2-4289-8e58-71942688b8ea\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") "
Nov 03 07:20:53.181390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.181366    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362d8b24-79e2-4289-8e58-71942688b8ea-kube-api-access-9jnkg\" (UniqueName: \"kubernetes.io/projected/362d8b24-79e2-4289-8e58-71942688b8ea-kube-api-access-9jnkg\") pod \"362d8b24-79e2-4289-8e58-71942688b8ea\" (UID: \"362d8b24-79e2-4289-8e58-71942688b8ea\") "
Nov 03 07:20:53.181636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.181607    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-openshift-ca" pod "362d8b24-79e2-4289-8e58-71942688b8ea" (UID: "362d8b24-79e2-4289-8e58-71942688b8ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:20:53.181847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.181826    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:53.183578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.183545    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362d8b24-79e2-4289-8e58-71942688b8ea-kube-api-access-9jnkg" pod "362d8b24-79e2-4289-8e58-71942688b8ea" (UID: "362d8b24-79e2-4289-8e58-71942688b8ea"). InnerVolumeSpecName "kube-api-access-9jnkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:20:53.183794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.183770    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-xks-ca" pod "362d8b24-79e2-4289-8e58-71942688b8ea" (UID: "362d8b24-79e2-4289-8e58-71942688b8ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:20:53.283412 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.283374    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jnkg\" (UniqueName: \"kubernetes.io/projected/362d8b24-79e2-4289-8e58-71942688b8ea-kube-api-access-9jnkg\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:53.283412 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.283406    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362d8b24-79e2-4289-8e58-71942688b8ea-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:20:53.880307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.880282    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"
Nov 03 07:20:53.880476 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.880314    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn" event={"ID":"362d8b24-79e2-4289-8e58-71942688b8ea","Type":"ContainerDied","Data":"08a3fcf0f82a39472fddf447f834eb3281c3605631fe23c79a084bc119650226"}
Nov 03 07:20:53.880476 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:53.880364    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08a3fcf0f82a39472fddf447f834eb3281c3605631fe23c79a084bc119650226"
Nov 03 07:20:55.341572 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:55.341538    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:55.365976 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:55.365685    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:55.940156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:55.940112    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:20:57.068719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.068688    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"]
Nov 03 07:20:57.117875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.117846    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05473694-698kj"]
Nov 03 07:20:57.745157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.745101    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"]
Nov 03 07:20:57.745706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.745693    4470 cpu_manager.go:414] "containerMap: removing container" podUID="362d8b24-79e2-4289-8e58-71942688b8ea" containerName="revoke-keys"
Nov 03 07:20:57.745756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.745707    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362d8b24-79e2-4289-8e58-71942688b8ea" containerName="revoke-keys"
Nov 03 07:20:57.745826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.745817    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="362d8b24-79e2-4289-8e58-71942688b8ea" containerName="revoke-keys"
Nov 03 07:20:57.749248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.749232    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:20:57.806091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.806058    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="166eb2cf-6201-47ae-a18c-7ef14098bf4b" path="/var/lib/kubelet/pods/166eb2cf-6201-47ae-a18c-7ef14098bf4b/volumes"
Nov 03 07:20:57.812886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.812862    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"]
Nov 03 07:20:57.857329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.857300    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:20:57.857479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.857348    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:20:57.857479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.857381    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:20:57.857479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.857428    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:20:57.896567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.896538    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f64454-xtrg6\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:20:57.896747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.896575    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f64454-xtrg6\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:20:57.896747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.896621    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59t4z\" (UniqueName: \"kubernetes.io/projected/aed08894-3603-4c93-bba3-ebc17ae888cc-kube-api-access-59t4z\") pod \"maas-api-revoke-sub-f5f64454-xtrg6\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:20:57.998024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.997939    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f64454-xtrg6\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:20:57.998024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.997975    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f64454-xtrg6\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:20:57.998024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.998018    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59t4z\" (UniqueName: \"kubernetes.io/projected/aed08894-3603-4c93-bba3-ebc17ae888cc-kube-api-access-59t4z\") pod \"maas-api-revoke-sub-f5f64454-xtrg6\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:20:57.998639 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:57.998619    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f64454-xtrg6\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:20:58.000565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:58.000544    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f64454-xtrg6\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:20:58.066868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:58.066839    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59t4z\" (UniqueName: \"kubernetes.io/projected/aed08894-3603-4c93-bba3-ebc17ae888cc-kube-api-access-59t4z\") pod \"maas-api-revoke-sub-f5f64454-xtrg6\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:20:58.362974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:58.362940    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:20:58.751266 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:58.751234    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"]
Nov 03 07:20:58.751842 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:20:58.751813    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed08894_3603_4c93_bba3_ebc17ae888cc.slice/crio-1b2c7d20de474e18f9d2b6479042168c4edbbdb304d862f2eaf8d20289c5a2fa WatchSource:0}: Error finding container 1b2c7d20de474e18f9d2b6479042168c4edbbdb304d862f2eaf8d20289c5a2fa: Status 404 returned error can't find the container with id 1b2c7d20de474e18f9d2b6479042168c4edbbdb304d862f2eaf8d20289c5a2fa
Nov 03 07:20:58.943519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:58.943490    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6" event={"ID":"aed08894-3603-4c93-bba3-ebc17ae888cc","Type":"ContainerStarted","Data":"1b2c7d20de474e18f9d2b6479042168c4edbbdb304d862f2eaf8d20289c5a2fa"}
Nov 03 07:20:59.952911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:59.952825    4470 generic.go:356] "Generic (PLEG): container finished" podID="aed08894-3603-4c93-bba3-ebc17ae888cc" containerID="4d4b500dd8707211c72f3a78141f6ba9f396b815e0b342c6c17a6e45f0f65dab" exitCode=0
Nov 03 07:20:59.953333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:20:59.952945    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6" event={"ID":"aed08894-3603-4c93-bba3-ebc17ae888cc","Type":"ContainerDied","Data":"4d4b500dd8707211c72f3a78141f6ba9f396b815e0b342c6c17a6e45f0f65dab"}
Nov 03 07:21:01.109387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.109362    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:21:01.133802 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.133773    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-openshift-ca\") pod \"aed08894-3603-4c93-bba3-ebc17ae888cc\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") "
Nov 03 07:21:01.133963 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.133898    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed08894-3603-4c93-bba3-ebc17ae888cc-kube-api-access-59t4z\" (UniqueName: \"kubernetes.io/projected/aed08894-3603-4c93-bba3-ebc17ae888cc-kube-api-access-59t4z\") pod \"aed08894-3603-4c93-bba3-ebc17ae888cc\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") "
Nov 03 07:21:01.134011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.133962    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-xks-ca\") pod \"aed08894-3603-4c93-bba3-ebc17ae888cc\" (UID: \"aed08894-3603-4c93-bba3-ebc17ae888cc\") "
Nov 03 07:21:01.134148 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.134113    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-openshift-ca" pod "aed08894-3603-4c93-bba3-ebc17ae888cc" (UID: "aed08894-3603-4c93-bba3-ebc17ae888cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:21:01.134460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.134441    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:01.136202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.136181    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-xks-ca" pod "aed08894-3603-4c93-bba3-ebc17ae888cc" (UID: "aed08894-3603-4c93-bba3-ebc17ae888cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:21:01.136305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.136285    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed08894-3603-4c93-bba3-ebc17ae888cc-kube-api-access-59t4z" pod "aed08894-3603-4c93-bba3-ebc17ae888cc" (UID: "aed08894-3603-4c93-bba3-ebc17ae888cc"). InnerVolumeSpecName "kube-api-access-59t4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:21:01.236240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.236161    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59t4z\" (UniqueName: \"kubernetes.io/projected/aed08894-3603-4c93-bba3-ebc17ae888cc-kube-api-access-59t4z\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:01.236240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.236197    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed08894-3603-4c93-bba3-ebc17ae888cc-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:01.968652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.968625    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"
Nov 03 07:21:01.968652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.968643    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6" event={"ID":"aed08894-3603-4c93-bba3-ebc17ae888cc","Type":"ContainerDied","Data":"1b2c7d20de474e18f9d2b6479042168c4edbbdb304d862f2eaf8d20289c5a2fa"}
Nov 03 07:21:01.968850 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:01.968686    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b2c7d20de474e18f9d2b6479042168c4edbbdb304d862f2eaf8d20289c5a2fa"
Nov 03 07:21:02.056998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:02.056963    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"]
Nov 03 07:21:02.062310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:02.062285    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaedccdc-mhzhm"]
Nov 03 07:21:03.814324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:03.814291    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1" path="/var/lib/kubelet/pods/f7e1c2a0-e24c-46d3-a0a9-bd4af7ea33f1/volumes"
Nov 03 07:21:04.759702 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:21:04.759663    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:21:07.988918 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:07.988892    4470 scope.go:122] "RemoveContainer" containerID="3f37625371d91f33cbfe38241389cb22f887ba8224981d47ee75b69a39815918"
Nov 03 07:21:12.666443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.666405    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"]
Nov 03 07:21:12.666989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.666975    4470 cpu_manager.go:414] "containerMap: removing container" podUID="aed08894-3603-4c93-bba3-ebc17ae888cc" containerName="revoke-keys"
Nov 03 07:21:12.667033 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.666990    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed08894-3603-4c93-bba3-ebc17ae888cc" containerName="revoke-keys"
Nov 03 07:21:12.667101 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.667092    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed08894-3603-4c93-bba3-ebc17ae888cc" containerName="revoke-keys"
Nov 03 07:21:12.672019 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.672003    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:12.683702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.683677    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"]
Nov 03 07:21:12.727596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.727568    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:21:12.727792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.727660    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:21:12.727792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.727660    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:21:12.727792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.727677    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:21:12.769735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.769704    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db28b20-qpd2h\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:12.769903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.769761    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t7th\" (UniqueName: \"kubernetes.io/projected/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-kube-api-access-5t7th\") pod \"maas-api-revoke-sub-8db28b20-qpd2h\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:12.769957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.769939    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db28b20-qpd2h\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:12.871207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.871175    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t7th\" (UniqueName: \"kubernetes.io/projected/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-kube-api-access-5t7th\") pod \"maas-api-revoke-sub-8db28b20-qpd2h\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:12.871423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.871327    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db28b20-qpd2h\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:12.871423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.871402    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db28b20-qpd2h\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:12.872126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.872058    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db28b20-qpd2h\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:12.873857 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.873828    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db28b20-qpd2h\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:12.935835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.935763    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t7th\" (UniqueName: \"kubernetes.io/projected/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-kube-api-access-5t7th\") pod \"maas-api-revoke-sub-8db28b20-qpd2h\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:12.986546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:12.986513    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:13.372552 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:13.372522    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"]
Nov 03 07:21:13.372681 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:21:13.372662    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a73d109_b726_4ba4_97d3_bcb4c37e0f28.slice/crio-a1d7dbfd86daf1abc29a6859e3c7c3419f5507f129dc44e5f3dc94ebdb1d678a WatchSource:0}: Error finding container a1d7dbfd86daf1abc29a6859e3c7c3419f5507f129dc44e5f3dc94ebdb1d678a: Status 404 returned error can't find the container with id a1d7dbfd86daf1abc29a6859e3c7c3419f5507f129dc44e5f3dc94ebdb1d678a
Nov 03 07:21:14.054119 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.054044    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"]
Nov 03 07:21:14.058068 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.058046    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:14.062628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.062593    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h" event={"ID":"8a73d109-b726-4ba4-97d3-bcb4c37e0f28","Type":"ContainerStarted","Data":"bf784798dbed856eb9ce9ebf032f6559cc29fbc02ca7d429647e6d2f174ba639"}
Nov 03 07:21:14.062746 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.062653    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h" event={"ID":"8a73d109-b726-4ba4-97d3-bcb4c37e0f28","Type":"ContainerStarted","Data":"a1d7dbfd86daf1abc29a6859e3c7c3419f5507f129dc44e5f3dc94ebdb1d678a"}
Nov 03 07:21:14.117269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.117237    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"]
Nov 03 07:21:14.189540 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.189498    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2hz\" (UniqueName: \"kubernetes.io/projected/5979d46b-5f6e-4a03-aabb-631774d784f7-kube-api-access-fw2hz\") pod \"maas-api-revoke-sub-a342e23f-djsxf\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:14.189729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.189639    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a342e23f-djsxf\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:14.189908 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.189887    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a342e23f-djsxf\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:14.291193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.291123    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a342e23f-djsxf\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:14.291193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.291195    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2hz\" (UniqueName: \"kubernetes.io/projected/5979d46b-5f6e-4a03-aabb-631774d784f7-kube-api-access-fw2hz\") pod \"maas-api-revoke-sub-a342e23f-djsxf\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:14.291407 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.291245    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a342e23f-djsxf\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:14.291842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.291815    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a342e23f-djsxf\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:14.293810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.293786    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a342e23f-djsxf\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:14.298745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.298703    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h" podStartSLOduration=2.298691319 podStartE2EDuration="2.298691319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:14.297495446 +0000 UTC m=+1326.629864441" watchObservedRunningTime="2026-11-03 07:21:14.298691319 +0000 UTC m=+1326.631060312"
Nov 03 07:21:14.356154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.356111    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2hz\" (UniqueName: \"kubernetes.io/projected/5979d46b-5f6e-4a03-aabb-631774d784f7-kube-api-access-fw2hz\") pod \"maas-api-revoke-sub-a342e23f-djsxf\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:14.373016 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.372988    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:14.753304 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:21:14.753260    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5979d46b_5f6e_4a03_aabb_631774d784f7.slice/crio-344044d6eb02d9186087b60ba9169c7bba367177ed817521cda64dac003903a0 WatchSource:0}: Error finding container 344044d6eb02d9186087b60ba9169c7bba367177ed817521cda64dac003903a0: Status 404 returned error can't find the container with id 344044d6eb02d9186087b60ba9169c7bba367177ed817521cda64dac003903a0
Nov 03 07:21:14.753448 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:14.753411    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"]
Nov 03 07:21:15.078855 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:15.078811    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf" event={"ID":"5979d46b-5f6e-4a03-aabb-631774d784f7","Type":"ContainerStarted","Data":"10b8afa2804108df1ca5a4c59f93ae291ea29c55b8e9a9d724a2f430ee05c44f"}
Nov 03 07:21:15.079355 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:15.078911    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf" event={"ID":"5979d46b-5f6e-4a03-aabb-631774d784f7","Type":"ContainerStarted","Data":"344044d6eb02d9186087b60ba9169c7bba367177ed817521cda64dac003903a0"}
Nov 03 07:21:15.080795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:15.080766    4470 generic.go:356] "Generic (PLEG): container finished" podID="8a73d109-b726-4ba4-97d3-bcb4c37e0f28" containerID="bf784798dbed856eb9ce9ebf032f6559cc29fbc02ca7d429647e6d2f174ba639" exitCode=0
Nov 03 07:21:15.080900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:15.080845    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h" event={"ID":"8a73d109-b726-4ba4-97d3-bcb4c37e0f28","Type":"ContainerDied","Data":"bf784798dbed856eb9ce9ebf032f6559cc29fbc02ca7d429647e6d2f174ba639"}
Nov 03 07:21:15.200642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:15.200580    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf" podStartSLOduration=1.200566817 podStartE2EDuration="1.200566817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:15.198309 +0000 UTC m=+1327.530677994" watchObservedRunningTime="2026-11-03 07:21:15.200566817 +0000 UTC m=+1327.532935811"
Nov 03 07:21:16.089350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.089321    4470 generic.go:356] "Generic (PLEG): container finished" podID="5979d46b-5f6e-4a03-aabb-631774d784f7" containerID="10b8afa2804108df1ca5a4c59f93ae291ea29c55b8e9a9d724a2f430ee05c44f" exitCode=0
Nov 03 07:21:16.089751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.089412    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf" event={"ID":"5979d46b-5f6e-4a03-aabb-631774d784f7","Type":"ContainerDied","Data":"10b8afa2804108df1ca5a4c59f93ae291ea29c55b8e9a9d724a2f430ee05c44f"}
Nov 03 07:21:16.230271 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.230247    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:16.317041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.317007    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-xks-ca\") pod \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") "
Nov 03 07:21:16.317256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.317060    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-openshift-ca\") pod \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") "
Nov 03 07:21:16.317302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.317253    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-kube-api-access-5t7th\" (UniqueName: \"kubernetes.io/projected/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-kube-api-access-5t7th\") pod \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\" (UID: \"8a73d109-b726-4ba4-97d3-bcb4c37e0f28\") "
Nov 03 07:21:16.317496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.317467    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-openshift-ca" pod "8a73d109-b726-4ba4-97d3-bcb4c37e0f28" (UID: "8a73d109-b726-4ba4-97d3-bcb4c37e0f28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:21:16.317697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.317679    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:16.319459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.319433    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-xks-ca" pod "8a73d109-b726-4ba4-97d3-bcb4c37e0f28" (UID: "8a73d109-b726-4ba4-97d3-bcb4c37e0f28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:21:16.319459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.319436    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-kube-api-access-5t7th" pod "8a73d109-b726-4ba4-97d3-bcb4c37e0f28" (UID: "8a73d109-b726-4ba4-97d3-bcb4c37e0f28"). InnerVolumeSpecName "kube-api-access-5t7th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:21:16.418535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.418448    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t7th\" (UniqueName: \"kubernetes.io/projected/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-kube-api-access-5t7th\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:16.418535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:16.418479    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a73d109-b726-4ba4-97d3-bcb4c37e0f28-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:17.097793 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.097768    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"
Nov 03 07:21:17.098220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.097809    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h" event={"ID":"8a73d109-b726-4ba4-97d3-bcb4c37e0f28","Type":"ContainerDied","Data":"a1d7dbfd86daf1abc29a6859e3c7c3419f5507f129dc44e5f3dc94ebdb1d678a"}
Nov 03 07:21:17.098220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.097859    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1d7dbfd86daf1abc29a6859e3c7c3419f5507f129dc44e5f3dc94ebdb1d678a"
Nov 03 07:21:17.244333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.244310    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:17.330271 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.330244    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-openshift-ca\") pod \"5979d46b-5f6e-4a03-aabb-631774d784f7\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") "
Nov 03 07:21:17.330467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.330293    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5979d46b-5f6e-4a03-aabb-631774d784f7-kube-api-access-fw2hz\" (UniqueName: \"kubernetes.io/projected/5979d46b-5f6e-4a03-aabb-631774d784f7-kube-api-access-fw2hz\") pod \"5979d46b-5f6e-4a03-aabb-631774d784f7\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") "
Nov 03 07:21:17.330537 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.330466    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-xks-ca\") pod \"5979d46b-5f6e-4a03-aabb-631774d784f7\" (UID: \"5979d46b-5f6e-4a03-aabb-631774d784f7\") "
Nov 03 07:21:17.330654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.330632    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-openshift-ca" pod "5979d46b-5f6e-4a03-aabb-631774d784f7" (UID: "5979d46b-5f6e-4a03-aabb-631774d784f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:21:17.330943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.330928    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:17.332622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.332591    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5979d46b-5f6e-4a03-aabb-631774d784f7-kube-api-access-fw2hz" pod "5979d46b-5f6e-4a03-aabb-631774d784f7" (UID: "5979d46b-5f6e-4a03-aabb-631774d784f7"). InnerVolumeSpecName "kube-api-access-fw2hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:21:17.332720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.332641    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-xks-ca" pod "5979d46b-5f6e-4a03-aabb-631774d784f7" (UID: "5979d46b-5f6e-4a03-aabb-631774d784f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:21:17.431964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.431877    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5979d46b-5f6e-4a03-aabb-631774d784f7-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:17.431964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:17.431904    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2hz\" (UniqueName: \"kubernetes.io/projected/5979d46b-5f6e-4a03-aabb-631774d784f7-kube-api-access-fw2hz\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:18.105677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:18.105601    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"
Nov 03 07:21:18.105677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:18.105632    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf" event={"ID":"5979d46b-5f6e-4a03-aabb-631774d784f7","Type":"ContainerDied","Data":"344044d6eb02d9186087b60ba9169c7bba367177ed817521cda64dac003903a0"}
Nov 03 07:21:18.106088 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:18.105684    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="344044d6eb02d9186087b60ba9169c7bba367177ed817521cda64dac003903a0"
Nov 03 07:21:22.937007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.936976    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"]
Nov 03 07:21:22.937569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.937555    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8a73d109-b726-4ba4-97d3-bcb4c37e0f28" containerName="revoke-keys"
Nov 03 07:21:22.937616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.937570    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a73d109-b726-4ba4-97d3-bcb4c37e0f28" containerName="revoke-keys"
Nov 03 07:21:22.937616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.937581    4470 cpu_manager.go:414] "containerMap: removing container" podUID="5979d46b-5f6e-4a03-aabb-631774d784f7" containerName="revoke-keys"
Nov 03 07:21:22.937616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.937586    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5979d46b-5f6e-4a03-aabb-631774d784f7" containerName="revoke-keys"
Nov 03 07:21:22.937707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.937693    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a73d109-b726-4ba4-97d3-bcb4c37e0f28" containerName="revoke-keys"
Nov 03 07:21:22.937707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.937700    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="5979d46b-5f6e-4a03-aabb-631774d784f7" containerName="revoke-keys"
Nov 03 07:21:22.941571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.941550    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:22.987810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.987774    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"]
Nov 03 07:21:22.993172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.993148    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"]
Nov 03 07:21:22.993324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.993258    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:22.997046 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:22.997028    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.038633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.038600    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"]
Nov 03 07:21:23.043543 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.043497    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.080252 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.080225    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"]
Nov 03 07:21:23.085600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.085564    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"]
Nov 03 07:21:23.085751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.085686    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.090673 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.090644    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"]
Nov 03 07:21:23.101256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101223    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbm2r\" (UniqueName: \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-kube-api-access-lbm2r\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.101430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101270    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.101430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101307    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.101430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101339    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pzsb\" (UniqueName: \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-kube-api-access-5pzsb\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.101430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101381    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.101430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101404    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.101707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101430    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.101707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101478    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e10d85-28e0-4333-91c0-0bb95e661abe-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.101707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101515    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.101707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101546    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.101707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101600    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662121e8-340c-4638-9be4-de860e5cc823-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.101707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101625    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.101707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101657    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.101707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101695    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.102102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101735    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.102102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101759    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.102102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101795    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.102102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101831    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.102102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101903    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.102102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101951    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.102102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.101978    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.102102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.102026    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.102102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.102049    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662121e8-340c-4638-9be4-de860e5cc823-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.102102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.102080    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npp9\" (UniqueName: \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-kube-api-access-7npp9\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.102640 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.102177    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.102789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.102723    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.102924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.102794    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.112804 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.112774    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"]
Nov 03 07:21:23.120178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.118052    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-130a2d11-tgj7h"]
Nov 03 07:21:23.169665 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.169276    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"]
Nov 03 07:21:23.203921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.203820    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e10d85-28e0-4333-91c0-0bb95e661abe-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.203921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.203860    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.203921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.203883    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc30b05-f8f1-4c5c-9197-384960da56e9-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.203921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.203901    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.203921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.203924    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.204352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.203981    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.204352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204077    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662121e8-340c-4638-9be4-de860e5cc823-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.204352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204105    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.204352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204154    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.204352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204183    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.204352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204210    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.204352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204239    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.204352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204278    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.204352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204300    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.204352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204321    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.204352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204331    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.204903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204361    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.204903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204390    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.204903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204484    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.204903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204596    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.204903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204646    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.204903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204714    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e10d85-28e0-4333-91c0-0bb95e661abe-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.204903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204739    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.204903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204765    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.205373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.204962    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.205373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205023    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.205373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205068    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzxl8\" (UniqueName: \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-kube-api-access-fzxl8\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.205373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205226    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8qpj\" (UniqueName: \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-kube-api-access-c8qpj\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.205373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205269    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.205373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205270    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662121e8-340c-4638-9be4-de860e5cc823-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.205373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205304    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.205731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205402    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.205731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205448    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.205731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205477    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.205731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205562    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.205731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205583    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.205731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205604    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.205731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205627    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.205731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205643    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662121e8-340c-4638-9be4-de860e5cc823-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.205731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205662    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npp9\" (UniqueName: \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-kube-api-access-7npp9\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.205731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205684    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.205731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205707    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205748    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205767    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205783    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205817    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbm2r\" (UniqueName: \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-kube-api-access-lbm2r\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205832    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205848    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205863    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205880    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205880    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205898    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pzsb\" (UniqueName: \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-kube-api-access-5pzsb\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205957    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.205986    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.206016    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.206049    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.206359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.206336    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.207268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.206461    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.209534 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.207502    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.209534 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.207982    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.209534 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.208896    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.209732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.209665    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.210118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.210098    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.210509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.210485    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.210684 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.210661    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662121e8-340c-4638-9be4-de860e5cc823-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.210770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.210754    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.238432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.238400    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"]
Nov 03 07:21:23.275726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.275684    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npp9\" (UniqueName: \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-kube-api-access-7npp9\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.275900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.275688    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pzsb\" (UniqueName: \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-kube-api-access-5pzsb\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.276301 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.276279    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.276421 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.276352    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.278331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.278308    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.278744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.278726    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbm2r\" (UniqueName: \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-kube-api-access-lbm2r\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.295723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.295690    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"]
Nov 03 07:21:23.307654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.307620    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.307833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.307663    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.307833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.307704    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.307833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.307737    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:23.307833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.307743    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.307833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.307769    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzxl8\" (UniqueName: \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-kube-api-access-fzxl8\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.307833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.307799    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8qpj\" (UniqueName: \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-kube-api-access-c8qpj\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.307833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.307827    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.308346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.307860    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.308346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.307895    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.308346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.307975    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.308346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308029    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.308346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308053    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.308346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308127    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.308346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308212    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.308346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308242    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.308346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308312    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.308796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308362    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.308796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308384    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.308796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308550    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.308796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308596    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc30b05-f8f1-4c5c-9197-384960da56e9-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.308796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308643    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.308796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308677    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.309292 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308944    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.309292 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.308609    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.309533 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.309510    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.309813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.309788    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc30b05-f8f1-4c5c-9197-384960da56e9-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.310014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.309985    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.310678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.310652    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.310853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.310832    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.311817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.311770    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.312482 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.312165    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.313748 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.313725    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.314841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.314820    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:23.373998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.373592    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.374982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.374744    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8qpj\" (UniqueName: \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-kube-api-access-c8qpj\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.375827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.375781    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.376361 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.376320    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzxl8\" (UniqueName: \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-kube-api-access-fzxl8\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.408538 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.408511    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:23.555672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.555635    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:23.660874 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.660839    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:23.710227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.710198    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"]
Nov 03 07:21:23.711854 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:21:23.711829    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662121e8_340c_4638_9be4_de860e5cc823.slice/crio-3ecf9044d288e60eafd3a1ad45f1c090b81f73b6a340c7384a4411343e5d08a1 WatchSource:0}: Error finding container 3ecf9044d288e60eafd3a1ad45f1c090b81f73b6a340c7384a4411343e5d08a1: Status 404 returned error can't find the container with id 3ecf9044d288e60eafd3a1ad45f1c090b81f73b6a340c7384a4411343e5d08a1
Nov 03 07:21:23.714568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.714526    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:23.714657 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.714611    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:23.714697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.714663    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:23.757306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.757250    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"]
Nov 03 07:21:23.770495 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:21:23.770472    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e10d85_28e0_4333_91c0_0bb95e661abe.slice/crio-973e755ccd604bbf41a2fc2e980a5abee4b386f2fe09ca570252022adeed5396 WatchSource:0}: Error finding container 973e755ccd604bbf41a2fc2e980a5abee4b386f2fe09ca570252022adeed5396: Status 404 returned error can't find the container with id 973e755ccd604bbf41a2fc2e980a5abee4b386f2fe09ca570252022adeed5396
Nov 03 07:21:23.774254 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.774200    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:23.774361 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.774300    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:23.774361 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.774341    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:23.820195 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:21:23.818720    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc30b05_f8f1_4c5c_9197_384960da56e9.slice/crio-2010e1e6e3b7c4806a3be45befb61f4cf4053a1545d31cd9e3789a5b689a4446 WatchSource:0}: Error finding container 2010e1e6e3b7c4806a3be45befb61f4cf4053a1545d31cd9e3789a5b689a4446: Status 404 returned error can't find the container with id 2010e1e6e3b7c4806a3be45befb61f4cf4053a1545d31cd9e3789a5b689a4446
Nov 03 07:21:23.820709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.820252    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:23.820709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.820342    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:23.820709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.820386    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:23.826740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.826709    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9494ded-12c4-4754-91c8-6662ada16b8d" path="/var/lib/kubelet/pods/a9494ded-12c4-4754-91c8-6662ada16b8d/volumes"
Nov 03 07:21:23.828712 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.828686    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"]
Nov 03 07:21:23.954576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.954476    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"]
Nov 03 07:21:23.956822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.956165    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:23.956822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.956365    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:23.956822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:23.956418    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:24.071581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.071545    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"]
Nov 03 07:21:24.081190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.081145    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:24.081298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.081249    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:24.081357 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.081295    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:24.161297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.161260    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" event={"ID":"381d4897-ba0e-4795-86d4-45fc61fdfcb7","Type":"ContainerStarted","Data":"f1012e3539133080c62326bcd80048addf1c311096062aa1d58e50c9f476946c"}
Nov 03 07:21:24.164281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.164248    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" event={"ID":"662121e8-340c-4638-9be4-de860e5cc823","Type":"ContainerStarted","Data":"ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a"}
Nov 03 07:21:24.164394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.164305    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" event={"ID":"662121e8-340c-4638-9be4-de860e5cc823","Type":"ContainerStarted","Data":"3ecf9044d288e60eafd3a1ad45f1c090b81f73b6a340c7384a4411343e5d08a1"}
Nov 03 07:21:24.167793 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.167765    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" event={"ID":"2fc30b05-f8f1-4c5c-9197-384960da56e9","Type":"ContainerStarted","Data":"72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1"}
Nov 03 07:21:24.167986 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.167966    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" event={"ID":"2fc30b05-f8f1-4c5c-9197-384960da56e9","Type":"ContainerStarted","Data":"2010e1e6e3b7c4806a3be45befb61f4cf4053a1545d31cd9e3789a5b689a4446"}
Nov 03 07:21:24.170430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.170190    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" event={"ID":"40e10d85-28e0-4333-91c0-0bb95e661abe","Type":"ContainerStarted","Data":"fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2"}
Nov 03 07:21:24.170430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.170237    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" event={"ID":"40e10d85-28e0-4333-91c0-0bb95e661abe","Type":"ContainerStarted","Data":"973e755ccd604bbf41a2fc2e980a5abee4b386f2fe09ca570252022adeed5396"}
Nov 03 07:21:24.173906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.173339    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" event={"ID":"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9","Type":"ContainerStarted","Data":"c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214"}
Nov 03 07:21:24.173906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.173372    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" event={"ID":"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9","Type":"ContainerStarted","Data":"27279d36cb1614dbb7ba1ddcc1c57845e4c2526bd4a66c209f527bf702c73068"}
Nov 03 07:21:24.308458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.308370    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:24.309843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.309622    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podStartSLOduration=2.309605818 podStartE2EDuration="2.309605818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:24.301393176 +0000 UTC m=+1336.633762173" watchObservedRunningTime="2026-11-03 07:21:24.309605818 +0000 UTC m=+1336.641974823"
Nov 03 07:21:24.313837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.313653    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:24.313837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.313724    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:24.315936 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.315913    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:24.317873 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.317841    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:24.317989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.317909    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:24.409485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.409452    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:24.411053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.411023    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:24.411179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.411080    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:24.435460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.435396    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podStartSLOduration=2.43537703 podStartE2EDuration="2.43537703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:24.429303172 +0000 UTC m=+1336.761672219" watchObservedRunningTime="2026-11-03 07:21:24.43537703 +0000 UTC m=+1336.767746025"
Nov 03 07:21:24.559842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.559651    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:24.561478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.561426    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podStartSLOduration=1.5614119739999999 podStartE2EDuration="1.561411974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:24.559315805 +0000 UTC m=+1336.891684808" watchObservedRunningTime="2026-11-03 07:21:24.561411974 +0000 UTC m=+1336.893780969"
Nov 03 07:21:24.688353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:24.688300    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podStartSLOduration=2.688286603 podStartE2EDuration="2.688286603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:24.684106415 +0000 UTC m=+1337.016475409" watchObservedRunningTime="2026-11-03 07:21:24.688286603 +0000 UTC m=+1337.020655597"
Nov 03 07:21:25.243161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.239383    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" event={"ID":"381d4897-ba0e-4795-86d4-45fc61fdfcb7","Type":"ContainerStarted","Data":"db4ca98833575ec82957daba56192550d1b065c5060119843f7596987232395a"}
Nov 03 07:21:25.312201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.311591    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:25.312201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.311661    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:25.332401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.332363    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:25.332698 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.332666    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:25.414402 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.414276    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:25.414402 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.414338    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:25.565224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.564301    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 07:21:25.565224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.564384    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 07:21:25.673163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.665506    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:25.677158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.674279    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 07:21:25.677158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:25.674347    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 07:21:26.133161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.129249    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:26.290160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.289265    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:26.318336 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.313557    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:26.318336 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.313618    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:26.323159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.319803    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:26.323159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.319855    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:26.366159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.365806    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:26.409293 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.407206    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:26.413158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.411466    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:26.413158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.411520    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:26.563162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.559481    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 07:21:26.563162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.559554    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 07:21:26.590416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.590366    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:26.648461 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.648409    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:26.692346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.692249    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 07:21:26.692631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.692593    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 07:21:26.701161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.697626    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:26.755289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.755225    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:26.909160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.908477    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 07:21:26.947232 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.947111    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:26.971159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:26.963255    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:27.152160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.130217    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:21:27.152160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.130655    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:21:27.331894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.315256    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:27.331894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.331007    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:21:27.333045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.332699    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podStartSLOduration=5.332680283 podStartE2EDuration="5.332680283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:25.408339151 +0000 UTC m=+1337.740708145" watchObservedRunningTime="2026-11-03 07:21:27.332680283 +0000 UTC m=+1339.665049279"
Nov 03 07:21:27.393199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.383902    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:27.393199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.383977    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:27.399181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.393879    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:27.399181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.393938    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:27.423163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.422658    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:27.430242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.430208    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:27.430491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.428534    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:27.430956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.430933    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:21:27.578320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.578266    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:27.597262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.597036    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:27.603603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.597839    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:21:27.699162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.649593    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:27.725572 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.699586    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 07:21:27.725979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.725955    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:21:27.732188 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.701494    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:27.732762 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.732737    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:21:27.844172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.699626    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 07:21:27.844172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.748037    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 07:21:27.844172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.750917    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 07:21:27.844172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.756506    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 07:21:27.844172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.782672    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:21:27.921218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.914642    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:27.921218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.919269    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:27.995161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.994235    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:28.003156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:27.995412    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:28.221248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.176673    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:28.336169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.330466    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:28.340256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.340216    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:28.361280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.353399    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:28.384170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.381502    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:21:28.384170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.381573    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:21:28.384170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.381610    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:21:28.429034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.403856    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:28.429034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.403909    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:28.429034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.418163    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:28.429034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.418222    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:28.461806 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.461769    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:28.604159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.599967    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:28.604159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.601597    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 03 07:21:28.609951 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.609909    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:21:28.625359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.625306    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 07:21:28.626183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.625589    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 07:21:28.703835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.703802    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 07:21:28.713265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.713222    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 07:21:28.746166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.739520    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:28.751167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.747626    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:28.788161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.784320    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 07:21:28.833283 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.833235    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 03 07:21:28.956006 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:28.955908    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:29.001833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.001793    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 03 07:21:29.042244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.008426    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:29.197161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.180318    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:29.219313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.197409    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:21:29.317550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.317515    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:29.317817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.317786    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:29.381186 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.379177    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:29.381186 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.379232    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:29.399295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.396703    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:29.443383 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.443317    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:29.489811 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.489717    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded"
Nov 03 07:21:29.514818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.493582    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:21:29.532483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.532447    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:29.532748 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.532722    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:29.658160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.644905    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 07:21:29.705217 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.694700    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 07:21:29.705498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.705468    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 07:21:29.705631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.694234    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:29.705741 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.667661    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Nov 03 07:21:29.741356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.709044    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 07:21:29.753161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.667720    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 03 07:21:29.786157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.742239    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:29.798511 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.775625    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:29.799016 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:29.796466    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 07:21:30.031343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.021509    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:30.095162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.039248    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 07:21:30.208289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.208240    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:21:30.283717 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.243472    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:30.331195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.324187    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:30.331195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.324251    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:30.331195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.325578    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:30.331195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.325627    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:30.501160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.497884    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:30.570220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.514691    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:30.570220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.515373    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:30.624160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.620958    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:30.728459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.728411    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Nov 03 07:21:30.784160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.759250    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:30.882218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.850190    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:30.888393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.852673    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 07:21:30.889475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.852702    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 07:21:30.979364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.979325    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 07:21:30.979648 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:30.979618    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 07:21:31.054417 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.054374    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 07:21:31.130161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.060728    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 03 07:21:31.130161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.075729    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 07:21:31.130161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.124777    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 07:21:31.190081 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:21:31.173460    4470 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.372s"
Nov 03 07:21:31.214926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.214864    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:21:31.244212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.244169    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:31.300269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.300223    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": context deadline exceeded"
Nov 03 07:21:31.308335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.308303    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:31.308611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.308585    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:31.324586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.324546    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:31.324857 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.324830    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:31.409545 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.409508    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:31.409836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.409803    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:31.604160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.570299    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 07:21:31.604160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.590246    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 07:21:31.604160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.592152    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:31.604160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.593942    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Nov 03 07:21:31.630276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.630227    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:31.666589 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.666550    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 07:21:31.666892 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.666862    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 07:21:31.729274 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.729229    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Nov 03 07:21:31.784686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.784644    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:31.886789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.886695    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:31.906289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.906239    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:31.906935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:31.906903    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 07:21:32.077638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.077589    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:32.125909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.125835    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:32.226218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.220437    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:32.248172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.246250    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:32.317006 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.316968    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:32.317290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.317260    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:32.317603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.317584    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:32.317767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.317746    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:32.415286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.415251    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:32.415548 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.415520    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:32.557590 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.557502    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 07:21:32.557895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.557851    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 07:21:32.597288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.597243    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:32.662554 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.662517    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 07:21:32.676269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.676221    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 07:21:32.722159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.682653    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:21:32.746165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.743235    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:32.801160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.800763    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 03 07:21:32.928202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.906979    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:32.928202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.927718    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 07:21:32.928202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.928020    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 07:21:32.928529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.928240    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:21:32.928529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:32.928273    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:33.123170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.121492    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:33.123170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.121821    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 03 07:21:33.132185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.131328    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 03 07:21:33.329618 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.278717    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:21:33.329618 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.279575    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:33.361316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.345581    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:33.361316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.345641    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:33.361316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.347155    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:33.361316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.347194    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:33.413123 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.413091    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:33.413446 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.413417    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:33.541155 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.540833    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:33.557659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.557622    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 07:21:33.557943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.557895    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 07:21:33.609220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.605991    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:33.679259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.678551    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 07:21:33.679259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.678616    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 07:21:33.749160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.725237    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:33.749669 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.749637    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:33.824752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.824693    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:33.930592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.930486    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:33.931167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.931117    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:33.931582 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:33.931557    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:34.126166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.125607    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 07:21:34.148966 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.132741    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:34.210260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.210111    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"]
Nov 03 07:21:34.288297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.288246    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:34.288976 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.288535    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:21:34.346163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.320810    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:34.535845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.535747    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:34.537206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.383632    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:34.537431 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.537407    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:34.537567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.453550    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:34.537679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.537661    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:34.539974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.539947    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4878ee-rkh22"]
Nov 03 07:21:34.565569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.565534    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 07:21:34.566042 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.565922    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 07:21:34.606376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.606309    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Nov 03 07:21:34.691307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.691270    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 07:21:34.691875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.691847    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 07:21:34.729172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.728220    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:34.750259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.750216    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:34.826369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.826234    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:34.844676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.843474    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:21:34.896945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.896908    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:21:34.945350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.943218    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 07:21:34.945350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:34.943540    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 07:21:35.166127 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.166030    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:35.299489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.299443    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:21:35.390501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.390300    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:35.393753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.392873    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"]
Nov 03 07:21:35.424794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.418694    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 07:21:35.424794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.418757    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 07:21:35.424794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.419034    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:35.443810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.443770    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:21:35.452329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.452294    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 07:21:35.452624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.452592    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 07:21:35.452834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.452794    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:35.452925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.452860    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:35.478175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.473441    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:21:35.478175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.475359    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:21:35.478175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.476239    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:35.478175 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:21:35.473515    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:21:35.492351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.492308    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"]
Nov 03 07:21:35.518514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.518484    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:21:35.538253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.536159    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:21:35.538253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.537157    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:21:35.538253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.537603    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:21:35.538253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.537754    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:21:35.538253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.537858    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:21:35.552834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.552803    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:21:35.561309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.561271    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 07:21:35.561474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.561332    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 07:21:35.590099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.581318    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvwz6\" (UniqueName: \"kubernetes.io/projected/ff624b96-4bef-4485-83a7-5ee92bd594a9-kube-api-access-xvwz6\") pod \"maas-api-revoke-sub-f5ddb302-x9qww\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:35.590099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.581453    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ddb302-x9qww\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:35.590099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.581630    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ddb302-x9qww\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:35.601832 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.600391    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:21:35.602924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.602244    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:21:35.618484 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.617224    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:21:35.664444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.664412    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 07:21:35.664731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.664692    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 07:21:35.683501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.683370    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ddb302-x9qww\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:35.683888 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.683861    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvwz6\" (UniqueName: \"kubernetes.io/projected/ff624b96-4bef-4485-83a7-5ee92bd594a9-kube-api-access-xvwz6\") pod \"maas-api-revoke-sub-f5ddb302-x9qww\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:35.684103 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.684085    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ddb302-x9qww\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:35.700358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.688580    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ddb302-x9qww\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:35.700987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.700958    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ddb302-x9qww\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:35.775158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.774647    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvwz6\" (UniqueName: \"kubernetes.io/projected/ff624b96-4bef-4485-83a7-5ee92bd594a9-kube-api-access-xvwz6\") pod \"maas-api-revoke-sub-f5ddb302-x9qww\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:35.898674 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.898644    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367d6236-a725-4e4d-a86d-68980e9c2672" path="/var/lib/kubelet/pods/367d6236-a725-4e4d-a86d-68980e9c2672/volumes"
Nov 03 07:21:35.963199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:35.963094    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:36.381936 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:36.381900    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 07:21:36.384226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:36.384189    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 07:21:36.459337 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:36.441351    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"]
Nov 03 07:21:36.608161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:36.514412    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808bedea-8kscb"]
Nov 03 07:21:36.773207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:36.770771    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:36.775355 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:36.775111    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:36.823158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:36.807917    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:21:36.911635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:36.895855    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:36.911635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:36.897245    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:36.916176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:36.912218    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:21:37.013169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.000561    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:37.013169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.001876    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:37.014459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.013669    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:21:37.070889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.070840    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"]
Nov 03 07:21:37.079945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.079917    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:37.127162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.125424    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"]
Nov 03 07:21:37.144289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.144047    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rn5\" (UniqueName: \"kubernetes.io/projected/85abeaec-c2dd-43ac-b037-76e1e9a735f8-kube-api-access-p6rn5\") pod \"maas-api-revoke-sub-8bba0527-r8ttm\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:37.144289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.144193    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bba0527-r8ttm\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:37.144289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.144256    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bba0527-r8ttm\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:37.246794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.246189    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rn5\" (UniqueName: \"kubernetes.io/projected/85abeaec-c2dd-43ac-b037-76e1e9a735f8-kube-api-access-p6rn5\") pod \"maas-api-revoke-sub-8bba0527-r8ttm\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:37.246794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.246686    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bba0527-r8ttm\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:37.247585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.247527    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bba0527-r8ttm\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:37.247887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.247864    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bba0527-r8ttm\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:37.255044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.251768    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bba0527-r8ttm\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:37.255044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.252023    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:37.255044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.252688    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:37.255044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.254299    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:21:37.282550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.281748    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"]
Nov 03 07:21:37.318981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.318956    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rn5\" (UniqueName: \"kubernetes.io/projected/85abeaec-c2dd-43ac-b037-76e1e9a735f8-kube-api-access-p6rn5\") pod \"maas-api-revoke-sub-8bba0527-r8ttm\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:37.321120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.321066    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:37.321386 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.321367    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:37.322034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.322016    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:21:37.436599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.436563    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:37.672436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.672345    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww" event={"ID":"ff624b96-4bef-4485-83a7-5ee92bd594a9","Type":"ContainerStarted","Data":"72bf4a3c18429f832bfd8cd6b4750dccda975c57c3af5889215b5c873e2cbc2f"}
Nov 03 07:21:37.815174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.814402    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7d54c2-c364-4b6c-8eb6-283b277046d1" path="/var/lib/kubelet/pods/4f7d54c2-c364-4b6c-8eb6-283b277046d1/volumes"
Nov 03 07:21:37.847888 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:21:37.847857    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85abeaec_c2dd_43ac_b037_76e1e9a735f8.slice/crio-756baf4b1ab1a6c743dd1f567c752cdfca714ebd44838bb48f9751244e89f1b3 WatchSource:0}: Error finding container 756baf4b1ab1a6c743dd1f567c752cdfca714ebd44838bb48f9751244e89f1b3: Status 404 returned error can't find the container with id 756baf4b1ab1a6c743dd1f567c752cdfca714ebd44838bb48f9751244e89f1b3
Nov 03 07:21:37.864930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:37.864900    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"]
Nov 03 07:21:38.681120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:38.681086    4470 generic.go:356] "Generic (PLEG): container finished" podID="ff624b96-4bef-4485-83a7-5ee92bd594a9" containerID="b9462f35c428dfc922fe30d9205ca30c88e02b2c2f6dad4164b34f789c121b4e" exitCode=0
Nov 03 07:21:38.681294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:38.681169    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww" event={"ID":"ff624b96-4bef-4485-83a7-5ee92bd594a9","Type":"ContainerDied","Data":"b9462f35c428dfc922fe30d9205ca30c88e02b2c2f6dad4164b34f789c121b4e"}
Nov 03 07:21:38.682944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:38.682920    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm" event={"ID":"85abeaec-c2dd-43ac-b037-76e1e9a735f8","Type":"ContainerStarted","Data":"eed6524d7546f55eee74c85270dcc15e5bcebaf86667a04e9d051fc5ce655ea3"}
Nov 03 07:21:38.683065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:38.682963    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm" event={"ID":"85abeaec-c2dd-43ac-b037-76e1e9a735f8","Type":"ContainerStarted","Data":"756baf4b1ab1a6c743dd1f567c752cdfca714ebd44838bb48f9751244e89f1b3"}
Nov 03 07:21:38.926599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:38.926546    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm" podStartSLOduration=1.92652839 podStartE2EDuration="1.92652839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:38.924473382 +0000 UTC m=+1351.256842377" watchObservedRunningTime="2026-11-03 07:21:38.92652839 +0000 UTC m=+1351.258897384"
Nov 03 07:21:39.692179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.692114    4470 generic.go:356] "Generic (PLEG): container finished" podID="85abeaec-c2dd-43ac-b037-76e1e9a735f8" containerID="eed6524d7546f55eee74c85270dcc15e5bcebaf86667a04e9d051fc5ce655ea3" exitCode=0
Nov 03 07:21:39.692357 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.692202    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm" event={"ID":"85abeaec-c2dd-43ac-b037-76e1e9a735f8","Type":"ContainerDied","Data":"eed6524d7546f55eee74c85270dcc15e5bcebaf86667a04e9d051fc5ce655ea3"}
Nov 03 07:21:39.868063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.868040    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:39.881563 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.881537    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-openshift-ca\") pod \"ff624b96-4bef-4485-83a7-5ee92bd594a9\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") "
Nov 03 07:21:39.881755 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.881689    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff624b96-4bef-4485-83a7-5ee92bd594a9-kube-api-access-xvwz6\" (UniqueName: \"kubernetes.io/projected/ff624b96-4bef-4485-83a7-5ee92bd594a9-kube-api-access-xvwz6\") pod \"ff624b96-4bef-4485-83a7-5ee92bd594a9\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") "
Nov 03 07:21:39.881817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.881778    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-xks-ca\") pod \"ff624b96-4bef-4485-83a7-5ee92bd594a9\" (UID: \"ff624b96-4bef-4485-83a7-5ee92bd594a9\") "
Nov 03 07:21:39.881912 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.881880    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-openshift-ca" pod "ff624b96-4bef-4485-83a7-5ee92bd594a9" (UID: "ff624b96-4bef-4485-83a7-5ee92bd594a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:21:39.882434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.882340    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:39.883868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.883847    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-xks-ca" pod "ff624b96-4bef-4485-83a7-5ee92bd594a9" (UID: "ff624b96-4bef-4485-83a7-5ee92bd594a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:21:39.883965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.883952    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff624b96-4bef-4485-83a7-5ee92bd594a9-kube-api-access-xvwz6" pod "ff624b96-4bef-4485-83a7-5ee92bd594a9" (UID: "ff624b96-4bef-4485-83a7-5ee92bd594a9"). InnerVolumeSpecName "kube-api-access-xvwz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:21:39.983726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.983637    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff624b96-4bef-4485-83a7-5ee92bd594a9-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:39.983726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:39.983666    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvwz6\" (UniqueName: \"kubernetes.io/projected/ff624b96-4bef-4485-83a7-5ee92bd594a9-kube-api-access-xvwz6\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:40.058438 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.058402    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"]
Nov 03 07:21:40.063637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.063611    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0aadb8a-xnsz4"]
Nov 03 07:21:40.253543 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.253463    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"]
Nov 03 07:21:40.254117 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.254098    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ff624b96-4bef-4485-83a7-5ee92bd594a9" containerName="revoke-keys"
Nov 03 07:21:40.254117 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.254118    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff624b96-4bef-4485-83a7-5ee92bd594a9" containerName="revoke-keys"
Nov 03 07:21:40.254338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.254320    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff624b96-4bef-4485-83a7-5ee92bd594a9" containerName="revoke-keys"
Nov 03 07:21:40.257992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.257976    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:40.287089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.287061    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd4d0691-cj2bf\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:40.287233 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.287099    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s4bs\" (UniqueName: \"kubernetes.io/projected/d793c2d8-9346-4870-bdb2-2c93af03d30b-kube-api-access-2s4bs\") pod \"maas-api-revoke-sub-dd4d0691-cj2bf\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:40.287233 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.287154    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd4d0691-cj2bf\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:40.319097 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.319061    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"]
Nov 03 07:21:40.388960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.388931    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd4d0691-cj2bf\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:40.389160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.388976    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s4bs\" (UniqueName: \"kubernetes.io/projected/d793c2d8-9346-4870-bdb2-2c93af03d30b-kube-api-access-2s4bs\") pod \"maas-api-revoke-sub-dd4d0691-cj2bf\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:40.389160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.389103    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd4d0691-cj2bf\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:40.389660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.389635    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd4d0691-cj2bf\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:40.391521 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.391497    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd4d0691-cj2bf\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:40.456970 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.456934    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s4bs\" (UniqueName: \"kubernetes.io/projected/d793c2d8-9346-4870-bdb2-2c93af03d30b-kube-api-access-2s4bs\") pod \"maas-api-revoke-sub-dd4d0691-cj2bf\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:40.571906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.571873    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:40.702467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.702440    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"
Nov 03 07:21:40.703444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.703413    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww" event={"ID":"ff624b96-4bef-4485-83a7-5ee92bd594a9","Type":"ContainerDied","Data":"72bf4a3c18429f832bfd8cd6b4750dccda975c57c3af5889215b5c873e2cbc2f"}
Nov 03 07:21:40.703552 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.703470    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72bf4a3c18429f832bfd8cd6b4750dccda975c57c3af5889215b5c873e2cbc2f"
Nov 03 07:21:40.839298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.839271    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:40.894728 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.894703    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-openshift-ca\") pod \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") "
Nov 03 07:21:40.894875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.894853    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85abeaec-c2dd-43ac-b037-76e1e9a735f8-kube-api-access-p6rn5\" (UniqueName: \"kubernetes.io/projected/85abeaec-c2dd-43ac-b037-76e1e9a735f8-kube-api-access-p6rn5\") pod \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") "
Nov 03 07:21:40.894952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.894938    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-xks-ca\") pod \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\" (UID: \"85abeaec-c2dd-43ac-b037-76e1e9a735f8\") "
Nov 03 07:21:40.895110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.895087    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-openshift-ca" pod "85abeaec-c2dd-43ac-b037-76e1e9a735f8" (UID: "85abeaec-c2dd-43ac-b037-76e1e9a735f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:21:40.895759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.895701    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:40.897316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.897293    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85abeaec-c2dd-43ac-b037-76e1e9a735f8-kube-api-access-p6rn5" pod "85abeaec-c2dd-43ac-b037-76e1e9a735f8" (UID: "85abeaec-c2dd-43ac-b037-76e1e9a735f8"). InnerVolumeSpecName "kube-api-access-p6rn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:21:40.897316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.897292    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-xks-ca" pod "85abeaec-c2dd-43ac-b037-76e1e9a735f8" (UID: "85abeaec-c2dd-43ac-b037-76e1e9a735f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:21:40.952504 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:21:40.952474    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd793c2d8_9346_4870_bdb2_2c93af03d30b.slice/crio-95699b2429608864282feba84d32464d763de14c2178841af20abf9235d99f7c WatchSource:0}: Error finding container 95699b2429608864282feba84d32464d763de14c2178841af20abf9235d99f7c: Status 404 returned error can't find the container with id 95699b2429608864282feba84d32464d763de14c2178841af20abf9235d99f7c
Nov 03 07:21:40.953594 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.953465    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"]
Nov 03 07:21:40.996609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.996573    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85abeaec-c2dd-43ac-b037-76e1e9a735f8-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:40.996609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:40.996599    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6rn5\" (UniqueName: \"kubernetes.io/projected/85abeaec-c2dd-43ac-b037-76e1e9a735f8-kube-api-access-p6rn5\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:41.710865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:41.710837    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"
Nov 03 07:21:41.710865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:41.710863    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm" event={"ID":"85abeaec-c2dd-43ac-b037-76e1e9a735f8","Type":"ContainerDied","Data":"756baf4b1ab1a6c743dd1f567c752cdfca714ebd44838bb48f9751244e89f1b3"}
Nov 03 07:21:41.711083 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:41.710928    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="756baf4b1ab1a6c743dd1f567c752cdfca714ebd44838bb48f9751244e89f1b3"
Nov 03 07:21:41.712665 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:41.712640    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf" event={"ID":"d793c2d8-9346-4870-bdb2-2c93af03d30b","Type":"ContainerStarted","Data":"3892f9452b98a85e88bdf55fb4475a43a0c4760bd4d5c94b7cb7fa5c379e0d96"}
Nov 03 07:21:41.712773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:41.712684    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf" event={"ID":"d793c2d8-9346-4870-bdb2-2c93af03d30b","Type":"ContainerStarted","Data":"95699b2429608864282feba84d32464d763de14c2178841af20abf9235d99f7c"}
Nov 03 07:21:41.808536 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:41.808506    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e9f928-ba59-4807-90a2-a02c4a9b26f2" path="/var/lib/kubelet/pods/e9e9f928-ba59-4807-90a2-a02c4a9b26f2/volumes"
Nov 03 07:21:41.835452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:41.835407    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf" podStartSLOduration=1.835394693 podStartE2EDuration="1.835394693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:41.833630603 +0000 UTC m=+1354.165999614" watchObservedRunningTime="2026-11-03 07:21:41.835394693 +0000 UTC m=+1354.167763687"
Nov 03 07:21:42.722107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:42.722073    4470 generic.go:356] "Generic (PLEG): container finished" podID="d793c2d8-9346-4870-bdb2-2c93af03d30b" containerID="3892f9452b98a85e88bdf55fb4475a43a0c4760bd4d5c94b7cb7fa5c379e0d96" exitCode=0
Nov 03 07:21:42.722527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:42.722163    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf" event={"ID":"d793c2d8-9346-4870-bdb2-2c93af03d30b","Type":"ContainerDied","Data":"3892f9452b98a85e88bdf55fb4475a43a0c4760bd4d5c94b7cb7fa5c379e0d96"}
Nov 03 07:21:43.898754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:43.898727    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:43.931502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:43.931462    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-openshift-ca\") pod \"d793c2d8-9346-4870-bdb2-2c93af03d30b\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") "
Nov 03 07:21:43.931660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:43.931580    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-xks-ca\") pod \"d793c2d8-9346-4870-bdb2-2c93af03d30b\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") "
Nov 03 07:21:43.931743 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:43.931714    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d793c2d8-9346-4870-bdb2-2c93af03d30b-kube-api-access-2s4bs\" (UniqueName: \"kubernetes.io/projected/d793c2d8-9346-4870-bdb2-2c93af03d30b-kube-api-access-2s4bs\") pod \"d793c2d8-9346-4870-bdb2-2c93af03d30b\" (UID: \"d793c2d8-9346-4870-bdb2-2c93af03d30b\") "
Nov 03 07:21:43.931825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:43.931784    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-openshift-ca" pod "d793c2d8-9346-4870-bdb2-2c93af03d30b" (UID: "d793c2d8-9346-4870-bdb2-2c93af03d30b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:21:43.932075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:43.932060    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:43.933794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:43.933761    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-xks-ca" pod "d793c2d8-9346-4870-bdb2-2c93af03d30b" (UID: "d793c2d8-9346-4870-bdb2-2c93af03d30b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:21:43.933901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:43.933879    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d793c2d8-9346-4870-bdb2-2c93af03d30b-kube-api-access-2s4bs" pod "d793c2d8-9346-4870-bdb2-2c93af03d30b" (UID: "d793c2d8-9346-4870-bdb2-2c93af03d30b"). InnerVolumeSpecName "kube-api-access-2s4bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:21:44.033584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:44.033506    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d793c2d8-9346-4870-bdb2-2c93af03d30b-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:44.033584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:44.033538    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s4bs\" (UniqueName: \"kubernetes.io/projected/d793c2d8-9346-4870-bdb2-2c93af03d30b-kube-api-access-2s4bs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:44.737637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:44.737607    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"
Nov 03 07:21:44.737800 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:44.737643    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf" event={"ID":"d793c2d8-9346-4870-bdb2-2c93af03d30b","Type":"ContainerDied","Data":"95699b2429608864282feba84d32464d763de14c2178841af20abf9235d99f7c"}
Nov 03 07:21:44.737800 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:44.737700    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95699b2429608864282feba84d32464d763de14c2178841af20abf9235d99f7c"
Nov 03 07:21:46.058562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:46.058523    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"]
Nov 03 07:21:46.111322 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:46.111292    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03eecb5b-5dsgc"]
Nov 03 07:21:47.059294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:47.059264    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"]
Nov 03 07:21:47.110107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:47.110077    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5088c2d-brfc5"]
Nov 03 07:21:47.807890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:47.807856    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8b46e6-f4f1-4d63-9414-43d1933129f6" path="/var/lib/kubelet/pods/0f8b46e6-f4f1-4d63-9414-43d1933129f6/volumes"
Nov 03 07:21:47.808460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:47.808445    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d11fe9-d84b-4ec3-b68f-8e199d12d417" path="/var/lib/kubelet/pods/11d11fe9-d84b-4ec3-b68f-8e199d12d417/volumes"
Nov 03 07:21:48.307350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.307320    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"]
Nov 03 07:21:48.307903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.307888    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d793c2d8-9346-4870-bdb2-2c93af03d30b" containerName="revoke-keys"
Nov 03 07:21:48.307956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.307904    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d793c2d8-9346-4870-bdb2-2c93af03d30b" containerName="revoke-keys"
Nov 03 07:21:48.307956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.307940    4470 cpu_manager.go:414] "containerMap: removing container" podUID="85abeaec-c2dd-43ac-b037-76e1e9a735f8" containerName="revoke-keys"
Nov 03 07:21:48.307956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.307949    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85abeaec-c2dd-43ac-b037-76e1e9a735f8" containerName="revoke-keys"
Nov 03 07:21:48.308082 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.308058    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d793c2d8-9346-4870-bdb2-2c93af03d30b" containerName="revoke-keys"
Nov 03 07:21:48.308082 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.308074    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="85abeaec-c2dd-43ac-b037-76e1e9a735f8" containerName="revoke-keys"
Nov 03 07:21:48.312841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.312825    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:48.372511 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.372481    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"]
Nov 03 07:21:48.389348 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.389318    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prmj\" (UniqueName: \"kubernetes.io/projected/9e1053cc-fd90-4e3e-bebb-b877635991b1-kube-api-access-9prmj\") pod \"maas-api-revoke-sub-8fb71197-w69pz\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:48.389538 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.389378    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb71197-w69pz\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:48.389538 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.389504    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb71197-w69pz\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:48.416925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.416896    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:21:48.416925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.416908    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:21:48.417143 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.417030    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:21:48.417967 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.417954    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:21:48.490488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.490457    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prmj\" (UniqueName: \"kubernetes.io/projected/9e1053cc-fd90-4e3e-bebb-b877635991b1-kube-api-access-9prmj\") pod \"maas-api-revoke-sub-8fb71197-w69pz\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:48.490667 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.490509    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb71197-w69pz\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:48.490667 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.490558    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb71197-w69pz\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:48.491270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.491244    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb71197-w69pz\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:48.493399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.493377    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb71197-w69pz\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:48.555413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.555389    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prmj\" (UniqueName: \"kubernetes.io/projected/9e1053cc-fd90-4e3e-bebb-b877635991b1-kube-api-access-9prmj\") pod \"maas-api-revoke-sub-8fb71197-w69pz\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:48.625826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:48.625798    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:49.017577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:49.017554    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"]
Nov 03 07:21:49.017887 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:21:49.017864    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1053cc_fd90_4e3e_bebb_b877635991b1.slice/crio-89ce9a7148a1324873fbf2aa8ea7db3ed96eb28a6353d4fd48165c2c3de0da5d WatchSource:0}: Error finding container 89ce9a7148a1324873fbf2aa8ea7db3ed96eb28a6353d4fd48165c2c3de0da5d: Status 404 returned error can't find the container with id 89ce9a7148a1324873fbf2aa8ea7db3ed96eb28a6353d4fd48165c2c3de0da5d
Nov 03 07:21:49.062160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:49.062115    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"]
Nov 03 07:21:49.076876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:49.076844    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c857612-85dnk"]
Nov 03 07:21:49.777729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:49.777629    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz" event={"ID":"9e1053cc-fd90-4e3e-bebb-b877635991b1","Type":"ContainerStarted","Data":"f3efac7ccb1ff4402e37f0c4180ab0534d6f830cbfa6b76ae9fff07be1805e47"}
Nov 03 07:21:49.777729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:49.777687    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz" event={"ID":"9e1053cc-fd90-4e3e-bebb-b877635991b1","Type":"ContainerStarted","Data":"89ce9a7148a1324873fbf2aa8ea7db3ed96eb28a6353d4fd48165c2c3de0da5d"}
Nov 03 07:21:49.814031 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:49.811011    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb12da53-2b74-4a64-aa6b-b1b5641e9542" path="/var/lib/kubelet/pods/fb12da53-2b74-4a64-aa6b-b1b5641e9542/volumes"
Nov 03 07:21:49.901381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:49.901325    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz" podStartSLOduration=1.9013122679999999 podStartE2EDuration="1.901312268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:49.89955098 +0000 UTC m=+1362.231919979" watchObservedRunningTime="2026-11-03 07:21:49.901312268 +0000 UTC m=+1362.233681261"
Nov 03 07:21:50.787441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:50.787405    4470 generic.go:356] "Generic (PLEG): container finished" podID="9e1053cc-fd90-4e3e-bebb-b877635991b1" containerID="f3efac7ccb1ff4402e37f0c4180ab0534d6f830cbfa6b76ae9fff07be1805e47" exitCode=0
Nov 03 07:21:50.787928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:50.787486    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz" event={"ID":"9e1053cc-fd90-4e3e-bebb-b877635991b1","Type":"ContainerDied","Data":"f3efac7ccb1ff4402e37f0c4180ab0534d6f830cbfa6b76ae9fff07be1805e47"}
Nov 03 07:21:51.974426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:51.974402    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:52.033370 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.033340    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-openshift-ca\") pod \"9e1053cc-fd90-4e3e-bebb-b877635991b1\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") "
Nov 03 07:21:52.033525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.033442    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-xks-ca\") pod \"9e1053cc-fd90-4e3e-bebb-b877635991b1\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") "
Nov 03 07:21:52.033569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.033521    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1053cc-fd90-4e3e-bebb-b877635991b1-kube-api-access-9prmj\" (UniqueName: \"kubernetes.io/projected/9e1053cc-fd90-4e3e-bebb-b877635991b1-kube-api-access-9prmj\") pod \"9e1053cc-fd90-4e3e-bebb-b877635991b1\" (UID: \"9e1053cc-fd90-4e3e-bebb-b877635991b1\") "
Nov 03 07:21:52.033718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.033695    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-openshift-ca" pod "9e1053cc-fd90-4e3e-bebb-b877635991b1" (UID: "9e1053cc-fd90-4e3e-bebb-b877635991b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:21:52.034089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.034066    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:52.035613 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.035579    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-xks-ca" pod "9e1053cc-fd90-4e3e-bebb-b877635991b1" (UID: "9e1053cc-fd90-4e3e-bebb-b877635991b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:21:52.035748 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.035735    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1053cc-fd90-4e3e-bebb-b877635991b1-kube-api-access-9prmj" pod "9e1053cc-fd90-4e3e-bebb-b877635991b1" (UID: "9e1053cc-fd90-4e3e-bebb-b877635991b1"). InnerVolumeSpecName "kube-api-access-9prmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:21:52.083860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.083829    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"]
Nov 03 07:21:52.087602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.087576    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dacfa8f-vrz6w"]
Nov 03 07:21:52.135996 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.135948    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e1053cc-fd90-4e3e-bebb-b877635991b1-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:52.135996 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.135992    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9prmj\" (UniqueName: \"kubernetes.io/projected/9e1053cc-fd90-4e3e-bebb-b877635991b1-kube-api-access-9prmj\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:52.806473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.806448    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"
Nov 03 07:21:52.806663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.806479    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz" event={"ID":"9e1053cc-fd90-4e3e-bebb-b877635991b1","Type":"ContainerDied","Data":"89ce9a7148a1324873fbf2aa8ea7db3ed96eb28a6353d4fd48165c2c3de0da5d"}
Nov 03 07:21:52.806663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.806524    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89ce9a7148a1324873fbf2aa8ea7db3ed96eb28a6353d4fd48165c2c3de0da5d"
Nov 03 07:21:52.812259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.812230    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"]
Nov 03 07:21:52.812813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.812801    4470 cpu_manager.go:414] "containerMap: removing container" podUID="9e1053cc-fd90-4e3e-bebb-b877635991b1" containerName="revoke-keys"
Nov 03 07:21:52.812868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.812818    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1053cc-fd90-4e3e-bebb-b877635991b1" containerName="revoke-keys"
Nov 03 07:21:52.812957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.812945    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1053cc-fd90-4e3e-bebb-b877635991b1" containerName="revoke-keys"
Nov 03 07:21:52.818199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.818179    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:52.876204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.876175    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"]
Nov 03 07:21:52.922107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.922071    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:21:52.922107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.922108    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:21:52.922312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.922127    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:21:52.922312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.922215    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:21:52.950292 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.950253    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46d980b3-r79x9\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:52.950549 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.950528    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46d980b3-r79x9\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:52.950673 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:52.950657    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwf2h\" (UniqueName: \"kubernetes.io/projected/77ad0537-60f7-4845-921d-ef80d3e2e138-kube-api-access-qwf2h\") pod \"maas-api-revoke-sub-46d980b3-r79x9\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:53.051694 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.051661    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46d980b3-r79x9\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:53.052073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.051789    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46d980b3-r79x9\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:53.052073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.051840    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwf2h\" (UniqueName: \"kubernetes.io/projected/77ad0537-60f7-4845-921d-ef80d3e2e138-kube-api-access-qwf2h\") pod \"maas-api-revoke-sub-46d980b3-r79x9\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:53.052469 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.052446    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46d980b3-r79x9\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:53.054427 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.054404    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46d980b3-r79x9\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:53.115623 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.115583    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwf2h\" (UniqueName: \"kubernetes.io/projected/77ad0537-60f7-4845-921d-ef80d3e2e138-kube-api-access-qwf2h\") pod \"maas-api-revoke-sub-46d980b3-r79x9\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:53.131519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.131488    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:53.247065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.247032    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"]
Nov 03 07:21:53.252998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.252967    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.314745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.314715    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"]
Nov 03 07:21:53.356105 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.356074    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.356294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.356210    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.356294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.356251    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.356294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.356283    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.356484 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.356307    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkppq\" (UniqueName: \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-kube-api-access-rkppq\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.356537 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.356503    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.356589 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.356543    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5255b255-a1c9-4da0-b31f-016adf8c066b-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.356683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.356659    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.356747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.356709    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.457563 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.457483    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.457563 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.457526    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5255b255-a1c9-4da0-b31f-016adf8c066b-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.457766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.457580    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.457766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.457614    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.457766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.457668    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.457766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.457741    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.457947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.457789    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.457947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.457833    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.457947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.457855    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkppq\" (UniqueName: \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-kube-api-access-rkppq\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.458095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.458065    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.458263 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.458236    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.458478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.458395    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5255b255-a1c9-4da0-b31f-016adf8c066b-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.458478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.458511    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.458478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.458546    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.460201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.460182    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.460751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.460721    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.529702 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:21:53.529671    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ad0537_60f7_4845_921d_ef80d3e2e138.slice/crio-e1fc8dbf8e0218b0d80a48ab5bd73809ae960596a4e48397ad4717e5662ddd5e WatchSource:0}: Error finding container e1fc8dbf8e0218b0d80a48ab5bd73809ae960596a4e48397ad4717e5662ddd5e: Status 404 returned error can't find the container with id e1fc8dbf8e0218b0d80a48ab5bd73809ae960596a4e48397ad4717e5662ddd5e
Nov 03 07:21:53.531114 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.531085    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"]
Nov 03 07:21:53.532924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.532807    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.533242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.533221    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkppq\" (UniqueName: \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-kube-api-access-rkppq\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.571073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.571048    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:53.807376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.807347    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae326739-3144-425d-86c9-e1502b925c3e" path="/var/lib/kubelet/pods/ae326739-3144-425d-86c9-e1502b925c3e/volumes"
Nov 03 07:21:53.814658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:53.814628    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9" event={"ID":"77ad0537-60f7-4845-921d-ef80d3e2e138","Type":"ContainerStarted","Data":"e1fc8dbf8e0218b0d80a48ab5bd73809ae960596a4e48397ad4717e5662ddd5e"}
Nov 03 07:21:54.090700 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:21:54.090659    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5255b255_a1c9_4da0_b31f_016adf8c066b.slice/crio-685bb461f0563851d766e7b0d15a0a2def902fb7130deca5e49e4b5e4a5cd21f WatchSource:0}: Error finding container 685bb461f0563851d766e7b0d15a0a2def902fb7130deca5e49e4b5e4a5cd21f: Status 404 returned error can't find the container with id 685bb461f0563851d766e7b0d15a0a2def902fb7130deca5e49e4b5e4a5cd21f
Nov 03 07:21:54.093289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:54.093263    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"]
Nov 03 07:21:54.093609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:54.093576    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:54.093682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:54.093655    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:54.093741 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:54.093703    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:21:54.853159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:54.852661    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" event={"ID":"5255b255-a1c9-4da0-b31f-016adf8c066b","Type":"ContainerStarted","Data":"98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522"}
Nov 03 07:21:54.853159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:54.852727    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" event={"ID":"5255b255-a1c9-4da0-b31f-016adf8c066b","Type":"ContainerStarted","Data":"685bb461f0563851d766e7b0d15a0a2def902fb7130deca5e49e4b5e4a5cd21f"}
Nov 03 07:21:54.876158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:54.871298    4470 generic.go:356] "Generic (PLEG): container finished" podID="77ad0537-60f7-4845-921d-ef80d3e2e138" containerID="23aaf63144e9883dae2132c8201f0d92544513c8de53b357310080e4f069d42a" exitCode=0
Nov 03 07:21:54.876158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:54.871479    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9" event={"ID":"77ad0537-60f7-4845-921d-ef80d3e2e138","Type":"ContainerDied","Data":"23aaf63144e9883dae2132c8201f0d92544513c8de53b357310080e4f069d42a"}
Nov 03 07:21:55.002161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:55.001325    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" podStartSLOduration=2.001304176 podStartE2EDuration="2.001304176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:55.000521659 +0000 UTC m=+1367.332890680" watchObservedRunningTime="2026-11-03 07:21:55.001304176 +0000 UTC m=+1367.333673173"
Nov 03 07:21:55.575162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:55.574355    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:21:55.575162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:55.574490    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 07:21:55.575162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:55.574536    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" podUID="5255b255-a1c9-4da0-b31f-016adf8c066b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 07:21:56.102319 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.102258    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"]
Nov 03 07:21:56.109697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.108968    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affe0c8f-5l8ft"]
Nov 03 07:21:56.125620 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.125584    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"]
Nov 03 07:21:56.133413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.133385    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba24433-hp2dx"]
Nov 03 07:21:56.188906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.188466    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:56.271866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.271828    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-openshift-ca\") pod \"77ad0537-60f7-4845-921d-ef80d3e2e138\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") "
Nov 03 07:21:56.272070 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.271930    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-xks-ca\") pod \"77ad0537-60f7-4845-921d-ef80d3e2e138\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") "
Nov 03 07:21:56.272070 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.271973    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ad0537-60f7-4845-921d-ef80d3e2e138-kube-api-access-qwf2h\" (UniqueName: \"kubernetes.io/projected/77ad0537-60f7-4845-921d-ef80d3e2e138-kube-api-access-qwf2h\") pod \"77ad0537-60f7-4845-921d-ef80d3e2e138\" (UID: \"77ad0537-60f7-4845-921d-ef80d3e2e138\") "
Nov 03 07:21:56.272323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.272292    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-openshift-ca" pod "77ad0537-60f7-4845-921d-ef80d3e2e138" (UID: "77ad0537-60f7-4845-921d-ef80d3e2e138"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:21:56.272596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.272573    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:56.274493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.274471    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-xks-ca" pod "77ad0537-60f7-4845-921d-ef80d3e2e138" (UID: "77ad0537-60f7-4845-921d-ef80d3e2e138"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:21:56.274595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.274502    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ad0537-60f7-4845-921d-ef80d3e2e138-kube-api-access-qwf2h" pod "77ad0537-60f7-4845-921d-ef80d3e2e138" (UID: "77ad0537-60f7-4845-921d-ef80d3e2e138"). InnerVolumeSpecName "kube-api-access-qwf2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:21:56.377957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.377918    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ad0537-60f7-4845-921d-ef80d3e2e138-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:56.377957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.377956    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwf2h\" (UniqueName: \"kubernetes.io/projected/77ad0537-60f7-4845-921d-ef80d3e2e138-kube-api-access-qwf2h\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:21:56.571667 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.571621    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 07:21:56.571843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.571686    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" podUID="5255b255-a1c9-4da0-b31f-016adf8c066b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 07:21:56.914074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.913978    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9" event={"ID":"77ad0537-60f7-4845-921d-ef80d3e2e138","Type":"ContainerDied","Data":"e1fc8dbf8e0218b0d80a48ab5bd73809ae960596a4e48397ad4717e5662ddd5e"}
Nov 03 07:21:56.914074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.914004    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"
Nov 03 07:21:56.914074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:56.914033    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1fc8dbf8e0218b0d80a48ab5bd73809ae960596a4e48397ad4717e5662ddd5e"
Nov 03 07:21:57.076695 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:57.076663    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"]
Nov 03 07:21:57.080279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:57.080256    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dcd155a-zh7wg"]
Nov 03 07:21:57.571568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:57.571533    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 07:21:57.571788 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:57.571589    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" podUID="5255b255-a1c9-4da0-b31f-016adf8c066b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 07:21:57.808177 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:57.808115    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b6fde5-fcde-4f39-9aa6-7821141f37d5" path="/var/lib/kubelet/pods/15b6fde5-fcde-4f39-9aa6-7821141f37d5/volumes"
Nov 03 07:21:57.808573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:57.808559    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a33ee05-4ed7-4a92-a154-6f659e8854ae" path="/var/lib/kubelet/pods/3a33ee05-4ed7-4a92-a154-6f659e8854ae/volumes"
Nov 03 07:21:57.808946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:57.808934    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522620d2-f7b5-4eca-b973-60ec250831f4" path="/var/lib/kubelet/pods/522620d2-f7b5-4eca-b973-60ec250831f4/volumes"
Nov 03 07:21:58.571636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:58.571593    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 07:21:58.572051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:58.571654    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" podUID="5255b255-a1c9-4da0-b31f-016adf8c066b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 07:21:59.031422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.031339    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"]
Nov 03 07:21:59.032006 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.031990    4470 cpu_manager.go:414] "containerMap: removing container" podUID="77ad0537-60f7-4845-921d-ef80d3e2e138" containerName="revoke-keys"
Nov 03 07:21:59.032062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.032008    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ad0537-60f7-4845-921d-ef80d3e2e138" containerName="revoke-keys"
Nov 03 07:21:59.032160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.032124    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ad0537-60f7-4845-921d-ef80d3e2e138" containerName="revoke-keys"
Nov 03 07:21:59.036871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.036853    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:21:59.094710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.094681    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"]
Nov 03 07:21:59.140571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.140533    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:21:59.140571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.140542    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:21:59.140571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.140558    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:21:59.141539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.141527    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:21:59.216116 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.216082    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a923f3-fdhx8\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:21:59.216335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.216319    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fjcm\" (UniqueName: \"kubernetes.io/projected/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-kube-api-access-4fjcm\") pod \"maas-api-revoke-sub-c7a923f3-fdhx8\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:21:59.216393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.216354    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a923f3-fdhx8\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:21:59.318067 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.318039    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a923f3-fdhx8\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:21:59.318254 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.318202    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a923f3-fdhx8\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:21:59.318415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.318398    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fjcm\" (UniqueName: \"kubernetes.io/projected/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-kube-api-access-4fjcm\") pod \"maas-api-revoke-sub-c7a923f3-fdhx8\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:21:59.318777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.318753    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a923f3-fdhx8\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:21:59.320689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.320658    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a923f3-fdhx8\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:21:59.382719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.382682    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fjcm\" (UniqueName: \"kubernetes.io/projected/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-kube-api-access-4fjcm\") pod \"maas-api-revoke-sub-c7a923f3-fdhx8\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:21:59.572622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.572540    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 07:21:59.572622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.572594    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" podUID="5255b255-a1c9-4da0-b31f-016adf8c066b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 07:21:59.672551 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:21:59.672509    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:22:00.073766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:00.073740    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"]
Nov 03 07:22:00.076316 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:00.076289    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c6a0a8_a112_4ab6_8066_65ce2701d6b6.slice/crio-6ceee9d85df1f0288f783a2eac8053784df0bfd0e503e4956de0b0a34ac9794f WatchSource:0}: Error finding container 6ceee9d85df1f0288f783a2eac8053784df0bfd0e503e4956de0b0a34ac9794f: Status 404 returned error can't find the container with id 6ceee9d85df1f0288f783a2eac8053784df0bfd0e503e4956de0b0a34ac9794f
Nov 03 07:22:00.571781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:00.571603    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 07:22:00.571781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:00.571674    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" podUID="5255b255-a1c9-4da0-b31f-016adf8c066b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 07:22:00.947095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:00.947052    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8" event={"ID":"83c6a0a8-a112-4ab6-8066-65ce2701d6b6","Type":"ContainerStarted","Data":"ae2921e0998c4263bd328de4731a6a74e2fcaefc43abc80e9aa0b1ad12408245"}
Nov 03 07:22:00.947497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:00.947122    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8" event={"ID":"83c6a0a8-a112-4ab6-8066-65ce2701d6b6","Type":"ContainerStarted","Data":"6ceee9d85df1f0288f783a2eac8053784df0bfd0e503e4956de0b0a34ac9794f"}
Nov 03 07:22:01.068673 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:01.068604    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8" podStartSLOduration=3.068589034 podStartE2EDuration="3.068589034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:01.067275878 +0000 UTC m=+1373.399644873" watchObservedRunningTime="2026-11-03 07:22:01.068589034 +0000 UTC m=+1373.400958024"
Nov 03 07:22:01.938505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:01.938451    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:22:01.939307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:01.939284    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:22:01.940360 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:01.940339    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:22:01.956378 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:01.956352    4470 generic.go:356] "Generic (PLEG): container finished" podID="83c6a0a8-a112-4ab6-8066-65ce2701d6b6" containerID="ae2921e0998c4263bd328de4731a6a74e2fcaefc43abc80e9aa0b1ad12408245" exitCode=0
Nov 03 07:22:01.956762 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:01.956486    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8" event={"ID":"83c6a0a8-a112-4ab6-8066-65ce2701d6b6","Type":"ContainerDied","Data":"ae2921e0998c4263bd328de4731a6a74e2fcaefc43abc80e9aa0b1ad12408245"}
Nov 03 07:22:02.056713 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.056678    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"]
Nov 03 07:22:02.061031 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.061008    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bf2752-kd8tt"]
Nov 03 07:22:02.173573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.173544    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"]
Nov 03 07:22:02.177828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.177802    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:02.238873 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.238778    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"]
Nov 03 07:22:02.255075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.255045    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52341cc7-pgvxz\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:02.255394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.255088    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhllz\" (UniqueName: \"kubernetes.io/projected/57750d51-6e2b-4427-b9fa-b00346cf28a8-kube-api-access-nhllz\") pod \"maas-api-revoke-sub-52341cc7-pgvxz\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:02.255394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.255367    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52341cc7-pgvxz\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:02.356607 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.356575    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52341cc7-pgvxz\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:02.356810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.356676    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52341cc7-pgvxz\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:02.356810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.356703    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhllz\" (UniqueName: \"kubernetes.io/projected/57750d51-6e2b-4427-b9fa-b00346cf28a8-kube-api-access-nhllz\") pod \"maas-api-revoke-sub-52341cc7-pgvxz\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:02.357471 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.357394    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52341cc7-pgvxz\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:02.359107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.359085    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52341cc7-pgvxz\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:02.420901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.420869    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhllz\" (UniqueName: \"kubernetes.io/projected/57750d51-6e2b-4427-b9fa-b00346cf28a8-kube-api-access-nhllz\") pod \"maas-api-revoke-sub-52341cc7-pgvxz\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:02.492088 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.492003    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:02.889792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.880836    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"]
Nov 03 07:22:02.889792 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:02.887108    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57750d51_6e2b_4427_b9fa_b00346cf28a8.slice/crio-e838891f79b5bf4ea0b0516cafa34ac39ff8402ffcd4aa088f7eaeb73da1aac1 WatchSource:0}: Error finding container e838891f79b5bf4ea0b0516cafa34ac39ff8402ffcd4aa088f7eaeb73da1aac1: Status 404 returned error can't find the container with id e838891f79b5bf4ea0b0516cafa34ac39ff8402ffcd4aa088f7eaeb73da1aac1
Nov 03 07:22:02.966571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:02.966532    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz" event={"ID":"57750d51-6e2b-4427-b9fa-b00346cf28a8","Type":"ContainerStarted","Data":"e838891f79b5bf4ea0b0516cafa34ac39ff8402ffcd4aa088f7eaeb73da1aac1"}
Nov 03 07:22:03.095072 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.095048    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:22:03.166826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.166745    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-kube-api-access-4fjcm\" (UniqueName: \"kubernetes.io/projected/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-kube-api-access-4fjcm\") pod \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") "
Nov 03 07:22:03.166826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.166783    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-openshift-ca\") pod \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") "
Nov 03 07:22:03.167023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.166887    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-xks-ca\") pod \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\" (UID: \"83c6a0a8-a112-4ab6-8066-65ce2701d6b6\") "
Nov 03 07:22:03.167246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.167220    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-openshift-ca" pod "83c6a0a8-a112-4ab6-8066-65ce2701d6b6" (UID: "83c6a0a8-a112-4ab6-8066-65ce2701d6b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:03.169170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.169118    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-xks-ca" pod "83c6a0a8-a112-4ab6-8066-65ce2701d6b6" (UID: "83c6a0a8-a112-4ab6-8066-65ce2701d6b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:03.169170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.169150    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-kube-api-access-4fjcm" pod "83c6a0a8-a112-4ab6-8066-65ce2701d6b6" (UID: "83c6a0a8-a112-4ab6-8066-65ce2701d6b6"). InnerVolumeSpecName "kube-api-access-4fjcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:03.268576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.268540    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:03.268576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.268571    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fjcm\" (UniqueName: \"kubernetes.io/projected/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-kube-api-access-4fjcm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:03.268576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.268582    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c6a0a8-a112-4ab6-8066-65ce2701d6b6-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:03.749768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.749737    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"]
Nov 03 07:22:03.750451 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.750430    4470 cpu_manager.go:414] "containerMap: removing container" podUID="83c6a0a8-a112-4ab6-8066-65ce2701d6b6" containerName="revoke-keys"
Nov 03 07:22:03.750451 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.750449    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c6a0a8-a112-4ab6-8066-65ce2701d6b6" containerName="revoke-keys"
Nov 03 07:22:03.750611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.750554    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c6a0a8-a112-4ab6-8066-65ce2701d6b6" containerName="revoke-keys"
Nov 03 07:22:03.754064 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.754046    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:03.774865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.774840    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a58829d4-ldkkq\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:03.774988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.774904    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a58829d4-ldkkq\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:03.775164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.775105    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrdtp\" (UniqueName: \"kubernetes.io/projected/54c392bc-814d-4983-b3b0-ce8c23f8ac95-kube-api-access-rrdtp\") pod \"maas-api-revoke-sub-a58829d4-ldkkq\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:03.808721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.808679    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22df6623-5855-4e65-909a-669d291db280" path="/var/lib/kubelet/pods/22df6623-5855-4e65-909a-669d291db280/volumes"
Nov 03 07:22:03.813318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.813288    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"]
Nov 03 07:22:03.876166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.876106    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdtp\" (UniqueName: \"kubernetes.io/projected/54c392bc-814d-4983-b3b0-ce8c23f8ac95-kube-api-access-rrdtp\") pod \"maas-api-revoke-sub-a58829d4-ldkkq\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:03.876324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.876306    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a58829d4-ldkkq\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:03.876410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.876397    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a58829d4-ldkkq\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:03.877038 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.877017    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a58829d4-ldkkq\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:03.878714 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.878696    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a58829d4-ldkkq\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:03.940078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.940049    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrdtp\" (UniqueName: \"kubernetes.io/projected/54c392bc-814d-4983-b3b0-ce8c23f8ac95-kube-api-access-rrdtp\") pod \"maas-api-revoke-sub-a58829d4-ldkkq\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:03.976412 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.976382    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8" event={"ID":"83c6a0a8-a112-4ab6-8066-65ce2701d6b6","Type":"ContainerDied","Data":"6ceee9d85df1f0288f783a2eac8053784df0bfd0e503e4956de0b0a34ac9794f"}
Nov 03 07:22:03.976886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.976431    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ceee9d85df1f0288f783a2eac8053784df0bfd0e503e4956de0b0a34ac9794f"
Nov 03 07:22:03.976886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.976388    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"
Nov 03 07:22:03.978657 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.978634    4470 generic.go:356] "Generic (PLEG): container finished" podID="57750d51-6e2b-4427-b9fa-b00346cf28a8" containerID="ceecac0b89a40c9dc07502fdb5241aaec133a8b0cdb50dfe5bcd17aaaee492f9" exitCode=0
Nov 03 07:22:03.978806 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:03.978767    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz" event={"ID":"57750d51-6e2b-4427-b9fa-b00346cf28a8","Type":"ContainerDied","Data":"ceecac0b89a40c9dc07502fdb5241aaec133a8b0cdb50dfe5bcd17aaaee492f9"}
Nov 03 07:22:04.067889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.067863    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:04.387481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.387411    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"]
Nov 03 07:22:04.392798 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.392778    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:04.452817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.452751    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"]
Nov 03 07:22:04.460691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.460664    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"]
Nov 03 07:22:04.462241 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:04.462202    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c392bc_814d_4983_b3b0_ce8c23f8ac95.slice/crio-790d396ba71f671184b1f37805eb82e98271e5c482594b3d5f6fcbe43eb87190 WatchSource:0}: Error finding container 790d396ba71f671184b1f37805eb82e98271e5c482594b3d5f6fcbe43eb87190: Status 404 returned error can't find the container with id 790d396ba71f671184b1f37805eb82e98271e5c482594b3d5f6fcbe43eb87190
Nov 03 07:22:04.484719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.484685    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06450ed2-snfq2\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:04.485633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.485610    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06450ed2-snfq2\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:04.485972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.485952    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7xx\" (UniqueName: \"kubernetes.io/projected/67b7fed8-56a7-473d-b5d1-a2848fd3748d-kube-api-access-4v7xx\") pod \"maas-api-revoke-sub-06450ed2-snfq2\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:04.587035 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.587004    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7xx\" (UniqueName: \"kubernetes.io/projected/67b7fed8-56a7-473d-b5d1-a2848fd3748d-kube-api-access-4v7xx\") pod \"maas-api-revoke-sub-06450ed2-snfq2\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:04.587230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.587119    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06450ed2-snfq2\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:04.587230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.587168    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06450ed2-snfq2\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:04.587791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.587759    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06450ed2-snfq2\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:04.589635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.589616    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06450ed2-snfq2\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:04.651689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.651618    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7xx\" (UniqueName: \"kubernetes.io/projected/67b7fed8-56a7-473d-b5d1-a2848fd3748d-kube-api-access-4v7xx\") pod \"maas-api-revoke-sub-06450ed2-snfq2\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:04.707169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.707101    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:04.765717 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:22:04.765357    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:22:04.988009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.987928    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq" event={"ID":"54c392bc-814d-4983-b3b0-ce8c23f8ac95","Type":"ContainerStarted","Data":"68ce5c7fb61cb1f56ee652d05b77338ab0635ede55e1ca66010c25297d33455f"}
Nov 03 07:22:04.988009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:04.987984    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq" event={"ID":"54c392bc-814d-4983-b3b0-ce8c23f8ac95","Type":"ContainerStarted","Data":"790d396ba71f671184b1f37805eb82e98271e5c482594b3d5f6fcbe43eb87190"}
Nov 03 07:22:05.095338 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:05.095304    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b7fed8_56a7_473d_b5d1_a2848fd3748d.slice/crio-66bfe94e387865f2a56d2e9ef2bfe2068662915144aaf21dba586aeed76ec4a2 WatchSource:0}: Error finding container 66bfe94e387865f2a56d2e9ef2bfe2068662915144aaf21dba586aeed76ec4a2: Status 404 returned error can't find the container with id 66bfe94e387865f2a56d2e9ef2bfe2068662915144aaf21dba586aeed76ec4a2
Nov 03 07:22:05.095861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.095836    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"]
Nov 03 07:22:05.112627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.112583    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq" podStartSLOduration=2.11256863 podStartE2EDuration="2.11256863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:05.108051111 +0000 UTC m=+1377.440420107" watchObservedRunningTime="2026-11-03 07:22:05.11256863 +0000 UTC m=+1377.444937637"
Nov 03 07:22:05.124366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.124344    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:05.195819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.195788    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57750d51-6e2b-4427-b9fa-b00346cf28a8-kube-api-access-nhllz\" (UniqueName: \"kubernetes.io/projected/57750d51-6e2b-4427-b9fa-b00346cf28a8-kube-api-access-nhllz\") pod \"57750d51-6e2b-4427-b9fa-b00346cf28a8\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") "
Nov 03 07:22:05.195990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.195874    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-xks-ca\") pod \"57750d51-6e2b-4427-b9fa-b00346cf28a8\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") "
Nov 03 07:22:05.195990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.195978    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-openshift-ca\") pod \"57750d51-6e2b-4427-b9fa-b00346cf28a8\" (UID: \"57750d51-6e2b-4427-b9fa-b00346cf28a8\") "
Nov 03 07:22:05.196455 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.196364    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-openshift-ca" pod "57750d51-6e2b-4427-b9fa-b00346cf28a8" (UID: "57750d51-6e2b-4427-b9fa-b00346cf28a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:05.196721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.196702    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:05.198039 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.198015    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-xks-ca" pod "57750d51-6e2b-4427-b9fa-b00346cf28a8" (UID: "57750d51-6e2b-4427-b9fa-b00346cf28a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:05.198150 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.198087    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57750d51-6e2b-4427-b9fa-b00346cf28a8-kube-api-access-nhllz" pod "57750d51-6e2b-4427-b9fa-b00346cf28a8" (UID: "57750d51-6e2b-4427-b9fa-b00346cf28a8"). InnerVolumeSpecName "kube-api-access-nhllz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:05.297864 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.297826    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhllz\" (UniqueName: \"kubernetes.io/projected/57750d51-6e2b-4427-b9fa-b00346cf28a8-kube-api-access-nhllz\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:05.297864 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.297861    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57750d51-6e2b-4427-b9fa-b00346cf28a8-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:05.996871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.996835    4470 generic.go:356] "Generic (PLEG): container finished" podID="54c392bc-814d-4983-b3b0-ce8c23f8ac95" containerID="68ce5c7fb61cb1f56ee652d05b77338ab0635ede55e1ca66010c25297d33455f" exitCode=0
Nov 03 07:22:05.997278 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.996916    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq" event={"ID":"54c392bc-814d-4983-b3b0-ce8c23f8ac95","Type":"ContainerDied","Data":"68ce5c7fb61cb1f56ee652d05b77338ab0635ede55e1ca66010c25297d33455f"}
Nov 03 07:22:05.998477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.998461    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"
Nov 03 07:22:05.998477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.998467    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz" event={"ID":"57750d51-6e2b-4427-b9fa-b00346cf28a8","Type":"ContainerDied","Data":"e838891f79b5bf4ea0b0516cafa34ac39ff8402ffcd4aa088f7eaeb73da1aac1"}
Nov 03 07:22:05.998643 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:05.998503    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e838891f79b5bf4ea0b0516cafa34ac39ff8402ffcd4aa088f7eaeb73da1aac1"
Nov 03 07:22:06.000253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:06.000235    4470 generic.go:356] "Generic (PLEG): container finished" podID="67b7fed8-56a7-473d-b5d1-a2848fd3748d" containerID="fa6d280236f8102ec78dc7cd43a688cb93e8890711d298993d43fdad1375a98f" exitCode=0
Nov 03 07:22:06.000337 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:06.000263    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2" event={"ID":"67b7fed8-56a7-473d-b5d1-a2848fd3748d","Type":"ContainerDied","Data":"fa6d280236f8102ec78dc7cd43a688cb93e8890711d298993d43fdad1375a98f"}
Nov 03 07:22:06.000337 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:06.000307    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2" event={"ID":"67b7fed8-56a7-473d-b5d1-a2848fd3748d","Type":"ContainerStarted","Data":"66bfe94e387865f2a56d2e9ef2bfe2068662915144aaf21dba586aeed76ec4a2"}
Nov 03 07:22:07.202062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.202035    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:07.244655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.244633    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:07.323653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.322908    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-openshift-ca\") pod \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") "
Nov 03 07:22:07.323653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.323012    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-xks-ca\") pod \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") "
Nov 03 07:22:07.323653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.323095    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-xks-ca\") pod \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") "
Nov 03 07:22:07.323653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.323125    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c392bc-814d-4983-b3b0-ce8c23f8ac95-kube-api-access-rrdtp\" (UniqueName: \"kubernetes.io/projected/54c392bc-814d-4983-b3b0-ce8c23f8ac95-kube-api-access-rrdtp\") pod \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") "
Nov 03 07:22:07.323653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.323683    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b7fed8-56a7-473d-b5d1-a2848fd3748d-kube-api-access-4v7xx\" (UniqueName: \"kubernetes.io/projected/67b7fed8-56a7-473d-b5d1-a2848fd3748d-kube-api-access-4v7xx\") pod \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\" (UID: \"67b7fed8-56a7-473d-b5d1-a2848fd3748d\") "
Nov 03 07:22:07.323653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.323887    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-openshift-ca\") pod \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\" (UID: \"54c392bc-814d-4983-b3b0-ce8c23f8ac95\") "
Nov 03 07:22:07.327212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.323936    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-openshift-ca" pod "67b7fed8-56a7-473d-b5d1-a2848fd3748d" (UID: "67b7fed8-56a7-473d-b5d1-a2848fd3748d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:07.327212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.325451    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-openshift-ca" pod "54c392bc-814d-4983-b3b0-ce8c23f8ac95" (UID: "54c392bc-814d-4983-b3b0-ce8c23f8ac95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:07.327212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.326006    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:07.327212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.326995    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-xks-ca" pod "67b7fed8-56a7-473d-b5d1-a2848fd3748d" (UID: "67b7fed8-56a7-473d-b5d1-a2848fd3748d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:07.328020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.327994    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-xks-ca" pod "54c392bc-814d-4983-b3b0-ce8c23f8ac95" (UID: "54c392bc-814d-4983-b3b0-ce8c23f8ac95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:07.328414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.328385    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c392bc-814d-4983-b3b0-ce8c23f8ac95-kube-api-access-rrdtp" pod "54c392bc-814d-4983-b3b0-ce8c23f8ac95" (UID: "54c392bc-814d-4983-b3b0-ce8c23f8ac95"). InnerVolumeSpecName "kube-api-access-rrdtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:07.329062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.329038    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b7fed8-56a7-473d-b5d1-a2848fd3748d-kube-api-access-4v7xx" pod "67b7fed8-56a7-473d-b5d1-a2848fd3748d" (UID: "67b7fed8-56a7-473d-b5d1-a2848fd3748d"). InnerVolumeSpecName "kube-api-access-4v7xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:07.426979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.426945    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:07.426979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.426973    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b7fed8-56a7-473d-b5d1-a2848fd3748d-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:07.426979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.426983    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c392bc-814d-4983-b3b0-ce8c23f8ac95-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:07.427276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.426992    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrdtp\" (UniqueName: \"kubernetes.io/projected/54c392bc-814d-4983-b3b0-ce8c23f8ac95-kube-api-access-rrdtp\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:07.427276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:07.427006    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v7xx\" (UniqueName: \"kubernetes.io/projected/67b7fed8-56a7-473d-b5d1-a2848fd3748d-kube-api-access-4v7xx\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:08.018750 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.018718    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"
Nov 03 07:22:08.018750 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.018738    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2" event={"ID":"67b7fed8-56a7-473d-b5d1-a2848fd3748d","Type":"ContainerDied","Data":"66bfe94e387865f2a56d2e9ef2bfe2068662915144aaf21dba586aeed76ec4a2"}
Nov 03 07:22:08.018997 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.018784    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66bfe94e387865f2a56d2e9ef2bfe2068662915144aaf21dba586aeed76ec4a2"
Nov 03 07:22:08.020315 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.020290    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq" event={"ID":"54c392bc-814d-4983-b3b0-ce8c23f8ac95","Type":"ContainerDied","Data":"790d396ba71f671184b1f37805eb82e98271e5c482594b3d5f6fcbe43eb87190"}
Nov 03 07:22:08.020438 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.020330    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"
Nov 03 07:22:08.020438 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.020334    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="790d396ba71f671184b1f37805eb82e98271e5c482594b3d5f6fcbe43eb87190"
Nov 03 07:22:08.026168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.026145    4470 scope.go:122] "RemoveContainer" containerID="707a7b321dc2f50aa51bb9c73dd6408d1159e46639ca9245ff11d6d27905e3e7"
Nov 03 07:22:08.036689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.036672    4470 scope.go:122] "RemoveContainer" containerID="86cbb4a9449692ed6333c7ca2c87172c241590d420d64e0666171d743cb9f727"
Nov 03 07:22:08.049346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.049327    4470 scope.go:122] "RemoveContainer" containerID="78ab9509a5e9486060afec73406e0ed8c103f2b61ae83117191e1d1eb842c7b0"
Nov 03 07:22:08.059174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.059127    4470 scope.go:122] "RemoveContainer" containerID="9fec5ac88e829020c4d4f91fe52d200420d273f2a7af17133ccfecd6cc5c4824"
Nov 03 07:22:08.068454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.068433    4470 scope.go:122] "RemoveContainer" containerID="e4896ba790824625f0e62e6560f476985f5ec9801a79eb78a8b0420661706050"
Nov 03 07:22:08.716924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.716893    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"]
Nov 03 07:22:08.717539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.717525    4470 cpu_manager.go:414] "containerMap: removing container" podUID="54c392bc-814d-4983-b3b0-ce8c23f8ac95" containerName="revoke-keys"
Nov 03 07:22:08.717586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.717540    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c392bc-814d-4983-b3b0-ce8c23f8ac95" containerName="revoke-keys"
Nov 03 07:22:08.717586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.717568    4470 cpu_manager.go:414] "containerMap: removing container" podUID="57750d51-6e2b-4427-b9fa-b00346cf28a8" containerName="revoke-keys"
Nov 03 07:22:08.717586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.717573    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57750d51-6e2b-4427-b9fa-b00346cf28a8" containerName="revoke-keys"
Nov 03 07:22:08.717672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.717599    4470 cpu_manager.go:414] "containerMap: removing container" podUID="67b7fed8-56a7-473d-b5d1-a2848fd3748d" containerName="revoke-keys"
Nov 03 07:22:08.717672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.717604    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b7fed8-56a7-473d-b5d1-a2848fd3748d" containerName="revoke-keys"
Nov 03 07:22:08.717733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.717711    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b7fed8-56a7-473d-b5d1-a2848fd3748d" containerName="revoke-keys"
Nov 03 07:22:08.717733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.717726    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c392bc-814d-4983-b3b0-ce8c23f8ac95" containerName="revoke-keys"
Nov 03 07:22:08.717796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.717734    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="57750d51-6e2b-4427-b9fa-b00346cf28a8" containerName="revoke-keys"
Nov 03 07:22:08.723408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.723389    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:08.779165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.779099    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:22:08.779523 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.779495    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"]
Nov 03 07:22:08.824647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.824617    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:22:08.824647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.824632    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:22:08.824647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.824617    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:22:08.846532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.846497    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bfj\" (UniqueName: \"kubernetes.io/projected/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-kube-api-access-56bfj\") pod \"maas-api-revoke-sub-4edb5170-xlhgq\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:08.846699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.846611    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4edb5170-xlhgq\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:08.846790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.846772    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4edb5170-xlhgq\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:08.948653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.948619    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4edb5170-xlhgq\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:08.948818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.948745    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bfj\" (UniqueName: \"kubernetes.io/projected/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-kube-api-access-56bfj\") pod \"maas-api-revoke-sub-4edb5170-xlhgq\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:08.948818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.948803    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4edb5170-xlhgq\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:08.949438 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.949414    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4edb5170-xlhgq\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:08.951163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:08.951119    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4edb5170-xlhgq\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:09.014644 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.014583    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bfj\" (UniqueName: \"kubernetes.io/projected/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-kube-api-access-56bfj\") pod \"maas-api-revoke-sub-4edb5170-xlhgq\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:09.038518 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.038493    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:09.419001 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:09.418955    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e66a022_ecb8_475d_b2c3_06a35e0e0da8.slice/crio-c5d1e06eb0ee4e3fc8c15c14fd627bb9d041331e42d1189f07e797cce4bb8ad0 WatchSource:0}: Error finding container c5d1e06eb0ee4e3fc8c15c14fd627bb9d041331e42d1189f07e797cce4bb8ad0: Status 404 returned error can't find the container with id c5d1e06eb0ee4e3fc8c15c14fd627bb9d041331e42d1189f07e797cce4bb8ad0
Nov 03 07:22:09.419517 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.419498    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"]
Nov 03 07:22:09.783062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.783034    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"]
Nov 03 07:22:09.787221 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.787204    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:09.847852 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.847816    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"]
Nov 03 07:22:09.860356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.860321    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ffd35b-ttfnb\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:09.860641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.860617    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxzwt\" (UniqueName: \"kubernetes.io/projected/c5a0ace7-837c-46b9-9412-0b62601b14c2-kube-api-access-fxzwt\") pod \"maas-api-revoke-sub-f2ffd35b-ttfnb\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:09.860731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.860663    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ffd35b-ttfnb\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:09.962152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.962060    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxzwt\" (UniqueName: \"kubernetes.io/projected/c5a0ace7-837c-46b9-9412-0b62601b14c2-kube-api-access-fxzwt\") pod \"maas-api-revoke-sub-f2ffd35b-ttfnb\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:09.962152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.962098    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ffd35b-ttfnb\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:09.962344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.962205    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ffd35b-ttfnb\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:09.962801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.962779    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ffd35b-ttfnb\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:09.964689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:09.964669    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ffd35b-ttfnb\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:10.026479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:10.026450    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxzwt\" (UniqueName: \"kubernetes.io/projected/c5a0ace7-837c-46b9-9412-0b62601b14c2-kube-api-access-fxzwt\") pod \"maas-api-revoke-sub-f2ffd35b-ttfnb\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:10.037760 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:10.037724    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq" event={"ID":"2e66a022-ecb8-475d-b2c3-06a35e0e0da8","Type":"ContainerStarted","Data":"2d72e097bf4c93fbeb8762a66eece611456176c12f0c71cc40b87b850050c4e0"}
Nov 03 07:22:10.037916 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:10.037808    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq" event={"ID":"2e66a022-ecb8-475d-b2c3-06a35e0e0da8","Type":"ContainerStarted","Data":"c5d1e06eb0ee4e3fc8c15c14fd627bb9d041331e42d1189f07e797cce4bb8ad0"}
Nov 03 07:22:10.102993 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:10.102964    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:10.162322 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:10.162271    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq" podStartSLOduration=2.162243929 podStartE2EDuration="2.162243929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:10.158521056 +0000 UTC m=+1382.490890054" watchObservedRunningTime="2026-11-03 07:22:10.162243929 +0000 UTC m=+1382.494612923"
Nov 03 07:22:10.486098 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:10.486059    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a0ace7_837c_46b9_9412_0b62601b14c2.slice/crio-d66d0d09b3e5e8a6c11ccd0ee0dc0cfb28c2fb7fba854383ef31a2fab54d183d WatchSource:0}: Error finding container d66d0d09b3e5e8a6c11ccd0ee0dc0cfb28c2fb7fba854383ef31a2fab54d183d: Status 404 returned error can't find the container with id d66d0d09b3e5e8a6c11ccd0ee0dc0cfb28c2fb7fba854383ef31a2fab54d183d
Nov 03 07:22:10.486244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:10.486222    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"]
Nov 03 07:22:11.047185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.047047    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb" event={"ID":"c5a0ace7-837c-46b9-9412-0b62601b14c2","Type":"ContainerStarted","Data":"cf46938ff0a2623485b1d85d4a9f62aff051d268f6492f476ab9027ddc1f4036"}
Nov 03 07:22:11.047185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.047108    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb" event={"ID":"c5a0ace7-837c-46b9-9412-0b62601b14c2","Type":"ContainerStarted","Data":"d66d0d09b3e5e8a6c11ccd0ee0dc0cfb28c2fb7fba854383ef31a2fab54d183d"}
Nov 03 07:22:11.048886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.048859    4470 generic.go:356] "Generic (PLEG): container finished" podID="2e66a022-ecb8-475d-b2c3-06a35e0e0da8" containerID="2d72e097bf4c93fbeb8762a66eece611456176c12f0c71cc40b87b850050c4e0" exitCode=0
Nov 03 07:22:11.048994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.048937    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq" event={"ID":"2e66a022-ecb8-475d-b2c3-06a35e0e0da8","Type":"ContainerDied","Data":"2d72e097bf4c93fbeb8762a66eece611456176c12f0c71cc40b87b850050c4e0"}
Nov 03 07:22:11.175535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.175473    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb" podStartSLOduration=2.175458751 podStartE2EDuration="2.175458751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:11.172814059 +0000 UTC m=+1383.505183053" watchObservedRunningTime="2026-11-03 07:22:11.175458751 +0000 UTC m=+1383.507827746"
Nov 03 07:22:11.539327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.539289    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"]
Nov 03 07:22:11.543535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.543518    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:11.604575 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.604496    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"]
Nov 03 07:22:11.687073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.687039    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8613f5c-fkh6r\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:11.687260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.687182    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chqt8\" (UniqueName: \"kubernetes.io/projected/830f0ede-e9b2-4b1f-ad01-9e9371230691-kube-api-access-chqt8\") pod \"maas-api-revoke-sub-f8613f5c-fkh6r\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:11.687307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.687273    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8613f5c-fkh6r\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:11.788625 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.788592    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chqt8\" (UniqueName: \"kubernetes.io/projected/830f0ede-e9b2-4b1f-ad01-9e9371230691-kube-api-access-chqt8\") pod \"maas-api-revoke-sub-f8613f5c-fkh6r\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:11.788809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.788661    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8613f5c-fkh6r\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:11.788892 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.788848    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8613f5c-fkh6r\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:11.789506 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.789483    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8613f5c-fkh6r\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:11.791163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.791105    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8613f5c-fkh6r\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:11.853681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.853647    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chqt8\" (UniqueName: \"kubernetes.io/projected/830f0ede-e9b2-4b1f-ad01-9e9371230691-kube-api-access-chqt8\") pod \"maas-api-revoke-sub-f8613f5c-fkh6r\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:11.856633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:11.856569    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:12.060924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.060891    4470 generic.go:356] "Generic (PLEG): container finished" podID="c5a0ace7-837c-46b9-9412-0b62601b14c2" containerID="cf46938ff0a2623485b1d85d4a9f62aff051d268f6492f476ab9027ddc1f4036" exitCode=0
Nov 03 07:22:12.061399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.060925    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb" event={"ID":"c5a0ace7-837c-46b9-9412-0b62601b14c2","Type":"ContainerDied","Data":"cf46938ff0a2623485b1d85d4a9f62aff051d268f6492f476ab9027ddc1f4036"}
Nov 03 07:22:12.216798 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.216771    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:12.295366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.295333    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-xks-ca\") pod \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") "
Nov 03 07:22:12.295559 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.295526    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-kube-api-access-56bfj\" (UniqueName: \"kubernetes.io/projected/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-kube-api-access-56bfj\") pod \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") "
Nov 03 07:22:12.295655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.295563    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-openshift-ca\") pod \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\" (UID: \"2e66a022-ecb8-475d-b2c3-06a35e0e0da8\") "
Nov 03 07:22:12.296234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.296189    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-openshift-ca" pod "2e66a022-ecb8-475d-b2c3-06a35e0e0da8" (UID: "2e66a022-ecb8-475d-b2c3-06a35e0e0da8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:12.299309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.299180    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-kube-api-access-56bfj" pod "2e66a022-ecb8-475d-b2c3-06a35e0e0da8" (UID: "2e66a022-ecb8-475d-b2c3-06a35e0e0da8"). InnerVolumeSpecName "kube-api-access-56bfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:12.299920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.299888    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-xks-ca" pod "2e66a022-ecb8-475d-b2c3-06a35e0e0da8" (UID: "2e66a022-ecb8-475d-b2c3-06a35e0e0da8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:12.300571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.300541    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56bfj\" (UniqueName: \"kubernetes.io/projected/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-kube-api-access-56bfj\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:12.300683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.300574    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:12.300683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.300589    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e66a022-ecb8-475d-b2c3-06a35e0e0da8-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:12.302482 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:12.302460    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830f0ede_e9b2_4b1f_ad01_9e9371230691.slice/crio-dbdbc8337c7e64ff7a56fff3e04eea58177a32bf1981f17b1ac03ec8ad9f121d WatchSource:0}: Error finding container dbdbc8337c7e64ff7a56fff3e04eea58177a32bf1981f17b1ac03ec8ad9f121d: Status 404 returned error can't find the container with id dbdbc8337c7e64ff7a56fff3e04eea58177a32bf1981f17b1ac03ec8ad9f121d
Nov 03 07:22:12.303905 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:12.303884    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"]
Nov 03 07:22:13.064140 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.064109    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"]
Nov 03 07:22:13.070979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.070954    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"
Nov 03 07:22:13.071177 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.070984    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq" event={"ID":"2e66a022-ecb8-475d-b2c3-06a35e0e0da8","Type":"ContainerDied","Data":"c5d1e06eb0ee4e3fc8c15c14fd627bb9d041331e42d1189f07e797cce4bb8ad0"}
Nov 03 07:22:13.071177 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.071031    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5d1e06eb0ee4e3fc8c15c14fd627bb9d041331e42d1189f07e797cce4bb8ad0"
Nov 03 07:22:13.072932 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.072908    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r" event={"ID":"830f0ede-e9b2-4b1f-ad01-9e9371230691","Type":"ContainerStarted","Data":"953c7ed42a9663ca4ec7d6ff1271c74bf7a96993eb967d6c82f113575e5048f8"}
Nov 03 07:22:13.073079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.072953    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r" event={"ID":"830f0ede-e9b2-4b1f-ad01-9e9371230691","Type":"ContainerStarted","Data":"dbdbc8337c7e64ff7a56fff3e04eea58177a32bf1981f17b1ac03ec8ad9f121d"}
Nov 03 07:22:13.114816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.114784    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"]
Nov 03 07:22:13.119078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.119059    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6c7915-f95bb"]
Nov 03 07:22:13.122719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.122694    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37d5456-wnwqw"]
Nov 03 07:22:13.197087 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.197029    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r" podStartSLOduration=2.197010437 podStartE2EDuration="2.197010437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:13.194857587 +0000 UTC m=+1385.527226580" watchObservedRunningTime="2026-11-03 07:22:13.197010437 +0000 UTC m=+1385.529379441"
Nov 03 07:22:13.225966 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.225942    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:13.313038 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.313011    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-openshift-ca\") pod \"c5a0ace7-837c-46b9-9412-0b62601b14c2\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") "
Nov 03 07:22:13.313240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.313097    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a0ace7-837c-46b9-9412-0b62601b14c2-kube-api-access-fxzwt\" (UniqueName: \"kubernetes.io/projected/c5a0ace7-837c-46b9-9412-0b62601b14c2-kube-api-access-fxzwt\") pod \"c5a0ace7-837c-46b9-9412-0b62601b14c2\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") "
Nov 03 07:22:13.313240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.313173    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-xks-ca\") pod \"c5a0ace7-837c-46b9-9412-0b62601b14c2\" (UID: \"c5a0ace7-837c-46b9-9412-0b62601b14c2\") "
Nov 03 07:22:13.313437 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.313415    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-openshift-ca" pod "c5a0ace7-837c-46b9-9412-0b62601b14c2" (UID: "c5a0ace7-837c-46b9-9412-0b62601b14c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:13.313633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.313615    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:13.315317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.315275    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-xks-ca" pod "c5a0ace7-837c-46b9-9412-0b62601b14c2" (UID: "c5a0ace7-837c-46b9-9412-0b62601b14c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:13.315424 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.315408    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a0ace7-837c-46b9-9412-0b62601b14c2-kube-api-access-fxzwt" pod "c5a0ace7-837c-46b9-9412-0b62601b14c2" (UID: "c5a0ace7-837c-46b9-9412-0b62601b14c2"). InnerVolumeSpecName "kube-api-access-fxzwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:13.414955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.414918    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxzwt\" (UniqueName: \"kubernetes.io/projected/c5a0ace7-837c-46b9-9412-0b62601b14c2-kube-api-access-fxzwt\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:13.414955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.414951    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a0ace7-837c-46b9-9412-0b62601b14c2-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:13.817601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.817570    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127a1ab7-123f-4276-9657-475cbadb8389" path="/var/lib/kubelet/pods/127a1ab7-123f-4276-9657-475cbadb8389/volumes"
Nov 03 07:22:13.818024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:13.818011    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88cb1aab-be65-43e4-aa18-8063a399efb0" path="/var/lib/kubelet/pods/88cb1aab-be65-43e4-aa18-8063a399efb0/volumes"
Nov 03 07:22:14.082356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.082261    4470 generic.go:356] "Generic (PLEG): container finished" podID="830f0ede-e9b2-4b1f-ad01-9e9371230691" containerID="953c7ed42a9663ca4ec7d6ff1271c74bf7a96993eb967d6c82f113575e5048f8" exitCode=0
Nov 03 07:22:14.082356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.082339    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r" event={"ID":"830f0ede-e9b2-4b1f-ad01-9e9371230691","Type":"ContainerDied","Data":"953c7ed42a9663ca4ec7d6ff1271c74bf7a96993eb967d6c82f113575e5048f8"}
Nov 03 07:22:14.083901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.083873    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb" event={"ID":"c5a0ace7-837c-46b9-9412-0b62601b14c2","Type":"ContainerDied","Data":"d66d0d09b3e5e8a6c11ccd0ee0dc0cfb28c2fb7fba854383ef31a2fab54d183d"}
Nov 03 07:22:14.083901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.083901    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"
Nov 03 07:22:14.084040 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.083914    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d66d0d09b3e5e8a6c11ccd0ee0dc0cfb28c2fb7fba854383ef31a2fab54d183d"
Nov 03 07:22:14.193637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.193601    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"]
Nov 03 07:22:14.194214 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.194200    4470 cpu_manager.go:414] "containerMap: removing container" podUID="2e66a022-ecb8-475d-b2c3-06a35e0e0da8" containerName="revoke-keys"
Nov 03 07:22:14.194265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.194215    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e66a022-ecb8-475d-b2c3-06a35e0e0da8" containerName="revoke-keys"
Nov 03 07:22:14.194265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.194236    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c5a0ace7-837c-46b9-9412-0b62601b14c2" containerName="revoke-keys"
Nov 03 07:22:14.194265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.194241    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a0ace7-837c-46b9-9412-0b62601b14c2" containerName="revoke-keys"
Nov 03 07:22:14.194356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.194346    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e66a022-ecb8-475d-b2c3-06a35e0e0da8" containerName="revoke-keys"
Nov 03 07:22:14.194387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.194357    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a0ace7-837c-46b9-9412-0b62601b14c2" containerName="revoke-keys"
Nov 03 07:22:14.199243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.199226    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:14.264727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.264693    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"]
Nov 03 07:22:14.326814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.326781    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4191ff27-a58d-406a-948b-638274197255-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ab93a6-287w5\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:14.326986 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.326837    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4191ff27-a58d-406a-948b-638274197255-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ab93a6-287w5\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:14.326986 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.326933    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvtw\" (UniqueName: \"kubernetes.io/projected/4191ff27-a58d-406a-948b-638274197255-kube-api-access-ptvtw\") pod \"maas-api-revoke-sub-e2ab93a6-287w5\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:14.427959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.427874    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4191ff27-a58d-406a-948b-638274197255-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ab93a6-287w5\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:14.427959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.427930    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4191ff27-a58d-406a-948b-638274197255-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ab93a6-287w5\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:14.428192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.428004    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvtw\" (UniqueName: \"kubernetes.io/projected/4191ff27-a58d-406a-948b-638274197255-kube-api-access-ptvtw\") pod \"maas-api-revoke-sub-e2ab93a6-287w5\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:14.428749 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.428729    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4191ff27-a58d-406a-948b-638274197255-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ab93a6-287w5\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:14.430388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.430369    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4191ff27-a58d-406a-948b-638274197255-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ab93a6-287w5\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:14.494514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.494484    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvtw\" (UniqueName: \"kubernetes.io/projected/4191ff27-a58d-406a-948b-638274197255-kube-api-access-ptvtw\") pod \"maas-api-revoke-sub-e2ab93a6-287w5\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:14.513806 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.513767    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:14.899700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:14.896778    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"]
Nov 03 07:22:14.900428 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:14.900400    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4191ff27_a58d_406a_948b_638274197255.slice/crio-788d3af4a6ec436a079c436e69685fa46e5c25dbfc33245b2e0738fdddfb1749 WatchSource:0}: Error finding container 788d3af4a6ec436a079c436e69685fa46e5c25dbfc33245b2e0738fdddfb1749: Status 404 returned error can't find the container with id 788d3af4a6ec436a079c436e69685fa46e5c25dbfc33245b2e0738fdddfb1749
Nov 03 07:22:15.093043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:15.093008    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5" event={"ID":"4191ff27-a58d-406a-948b-638274197255","Type":"ContainerStarted","Data":"788d3af4a6ec436a079c436e69685fa46e5c25dbfc33245b2e0738fdddfb1749"}
Nov 03 07:22:15.259007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:15.258983    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:15.340756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:15.340721    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-openshift-ca\") pod \"830f0ede-e9b2-4b1f-ad01-9e9371230691\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") "
Nov 03 07:22:15.340921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:15.340786    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-xks-ca\") pod \"830f0ede-e9b2-4b1f-ad01-9e9371230691\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") "
Nov 03 07:22:15.340921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:15.340897    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830f0ede-e9b2-4b1f-ad01-9e9371230691-kube-api-access-chqt8\" (UniqueName: \"kubernetes.io/projected/830f0ede-e9b2-4b1f-ad01-9e9371230691-kube-api-access-chqt8\") pod \"830f0ede-e9b2-4b1f-ad01-9e9371230691\" (UID: \"830f0ede-e9b2-4b1f-ad01-9e9371230691\") "
Nov 03 07:22:15.341093 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:15.341072    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-openshift-ca" pod "830f0ede-e9b2-4b1f-ad01-9e9371230691" (UID: "830f0ede-e9b2-4b1f-ad01-9e9371230691"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:15.341428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:15.341403    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:15.343224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:15.343198    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-xks-ca" pod "830f0ede-e9b2-4b1f-ad01-9e9371230691" (UID: "830f0ede-e9b2-4b1f-ad01-9e9371230691"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:15.343346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:15.343278    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830f0ede-e9b2-4b1f-ad01-9e9371230691-kube-api-access-chqt8" pod "830f0ede-e9b2-4b1f-ad01-9e9371230691" (UID: "830f0ede-e9b2-4b1f-ad01-9e9371230691"). InnerVolumeSpecName "kube-api-access-chqt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:15.443033 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:15.442952    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830f0ede-e9b2-4b1f-ad01-9e9371230691-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:15.443033 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:15.442983    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chqt8\" (UniqueName: \"kubernetes.io/projected/830f0ede-e9b2-4b1f-ad01-9e9371230691-kube-api-access-chqt8\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:16.101786 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:16.101697    4470 generic.go:356] "Generic (PLEG): container finished" podID="4191ff27-a58d-406a-948b-638274197255" containerID="406a06a9533f46374c554a5ec57805a22b68c21d96ea515540706f0305b2bbbf" exitCode=0
Nov 03 07:22:16.102260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:16.101789    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5" event={"ID":"4191ff27-a58d-406a-948b-638274197255","Type":"ContainerDied","Data":"406a06a9533f46374c554a5ec57805a22b68c21d96ea515540706f0305b2bbbf"}
Nov 03 07:22:16.103304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:16.103280    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r" event={"ID":"830f0ede-e9b2-4b1f-ad01-9e9371230691","Type":"ContainerDied","Data":"dbdbc8337c7e64ff7a56fff3e04eea58177a32bf1981f17b1ac03ec8ad9f121d"}
Nov 03 07:22:16.103397 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:16.103309    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"
Nov 03 07:22:16.103397 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:16.103324    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbdbc8337c7e64ff7a56fff3e04eea58177a32bf1981f17b1ac03ec8ad9f121d"
Nov 03 07:22:17.295085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:17.295049    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:17.470110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:17.470012    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4191ff27-a58d-406a-948b-638274197255-kube-api-access-ptvtw\" (UniqueName: \"kubernetes.io/projected/4191ff27-a58d-406a-948b-638274197255-kube-api-access-ptvtw\") pod \"4191ff27-a58d-406a-948b-638274197255\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") "
Nov 03 07:22:17.470304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:17.470197    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4191ff27-a58d-406a-948b-638274197255-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4191ff27-a58d-406a-948b-638274197255-maas-api-openshift-ca\") pod \"4191ff27-a58d-406a-948b-638274197255\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") "
Nov 03 07:22:17.470304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:17.470233    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4191ff27-a58d-406a-948b-638274197255-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4191ff27-a58d-406a-948b-638274197255-maas-api-xks-ca\") pod \"4191ff27-a58d-406a-948b-638274197255\" (UID: \"4191ff27-a58d-406a-948b-638274197255\") "
Nov 03 07:22:17.471065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:17.471032    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4191ff27-a58d-406a-948b-638274197255-maas-api-openshift-ca" pod "4191ff27-a58d-406a-948b-638274197255" (UID: "4191ff27-a58d-406a-948b-638274197255"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:17.473190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:17.473118    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4191ff27-a58d-406a-948b-638274197255-kube-api-access-ptvtw" pod "4191ff27-a58d-406a-948b-638274197255" (UID: "4191ff27-a58d-406a-948b-638274197255"). InnerVolumeSpecName "kube-api-access-ptvtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:17.473317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:17.473238    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4191ff27-a58d-406a-948b-638274197255-maas-api-xks-ca" pod "4191ff27-a58d-406a-948b-638274197255" (UID: "4191ff27-a58d-406a-948b-638274197255"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:17.572216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:17.572177    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptvtw\" (UniqueName: \"kubernetes.io/projected/4191ff27-a58d-406a-948b-638274197255-kube-api-access-ptvtw\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:17.572216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:17.572209    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4191ff27-a58d-406a-948b-638274197255-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:17.572216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:17.572219    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4191ff27-a58d-406a-948b-638274197255-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:18.121017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:18.120934    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"
Nov 03 07:22:18.121017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:18.120958    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5" event={"ID":"4191ff27-a58d-406a-948b-638274197255","Type":"ContainerDied","Data":"788d3af4a6ec436a079c436e69685fa46e5c25dbfc33245b2e0738fdddfb1749"}
Nov 03 07:22:18.121017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:18.121009    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="788d3af4a6ec436a079c436e69685fa46e5c25dbfc33245b2e0738fdddfb1749"
Nov 03 07:22:21.898519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.898483    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"]
Nov 03 07:22:21.899108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.899091    4470 cpu_manager.go:414] "containerMap: removing container" podUID="4191ff27-a58d-406a-948b-638274197255" containerName="revoke-keys"
Nov 03 07:22:21.899187 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.899110    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4191ff27-a58d-406a-948b-638274197255" containerName="revoke-keys"
Nov 03 07:22:21.899187 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.899125    4470 cpu_manager.go:414] "containerMap: removing container" podUID="830f0ede-e9b2-4b1f-ad01-9e9371230691" containerName="revoke-keys"
Nov 03 07:22:21.899187 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.899146    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830f0ede-e9b2-4b1f-ad01-9e9371230691" containerName="revoke-keys"
Nov 03 07:22:21.899294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.899274    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="4191ff27-a58d-406a-948b-638274197255" containerName="revoke-keys"
Nov 03 07:22:21.899294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.899283    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="830f0ede-e9b2-4b1f-ad01-9e9371230691" containerName="revoke-keys"
Nov 03 07:22:21.903015 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.902999    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:21.924147 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.924104    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18231cb1-jvzgs\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:21.924253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.924170    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rgf\" (UniqueName: \"kubernetes.io/projected/3003102d-67e5-480d-8f93-401e01e2f017-kube-api-access-t6rgf\") pod \"maas-api-revoke-sub-18231cb1-jvzgs\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:21.924353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.924333    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18231cb1-jvzgs\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:21.969630 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:21.969596    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"]
Nov 03 07:22:22.013841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.013805    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:22:22.013841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.013809    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:22:22.013841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.013805    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:22:22.014079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.013860    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:22:22.026012 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.025985    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18231cb1-jvzgs\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:22.026234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.026026    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rgf\" (UniqueName: \"kubernetes.io/projected/3003102d-67e5-480d-8f93-401e01e2f017-kube-api-access-t6rgf\") pod \"maas-api-revoke-sub-18231cb1-jvzgs\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:22.026359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.026259    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18231cb1-jvzgs\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:22.027350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.027328    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18231cb1-jvzgs\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:22.028833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.028813    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18231cb1-jvzgs\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:22.089900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.089879    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rgf\" (UniqueName: \"kubernetes.io/projected/3003102d-67e5-480d-8f93-401e01e2f017-kube-api-access-t6rgf\") pod \"maas-api-revoke-sub-18231cb1-jvzgs\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:22.217751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.217642    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:22.370494 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.370463    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"]
Nov 03 07:22:22.376048 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.376032    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:22.431875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.431840    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2xr\" (UniqueName: \"kubernetes.io/projected/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-kube-api-access-9d2xr\") pod \"maas-api-revoke-sub-4a07b01f-cd26q\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:22.432149 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.432105    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a07b01f-cd26q\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:22.432272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.432247    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a07b01f-cd26q\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:22.437170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.437107    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"]
Nov 03 07:22:22.534109 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.534014    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2xr\" (UniqueName: \"kubernetes.io/projected/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-kube-api-access-9d2xr\") pod \"maas-api-revoke-sub-4a07b01f-cd26q\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:22.534364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.534161    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a07b01f-cd26q\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:22.534364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.534212    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a07b01f-cd26q\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:22.534929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.534909    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a07b01f-cd26q\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:22.537199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.537179    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a07b01f-cd26q\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:22.598594 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.598560    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2xr\" (UniqueName: \"kubernetes.io/projected/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-kube-api-access-9d2xr\") pod \"maas-api-revoke-sub-4a07b01f-cd26q\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:22.613519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.613499    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"]
Nov 03 07:22:22.614513 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:22.614487    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3003102d_67e5_480d_8f93_401e01e2f017.slice/crio-96101031d4d65939894ad39fa928e44ef51c640fe366c4d6b1d799ed4eb2abab WatchSource:0}: Error finding container 96101031d4d65939894ad39fa928e44ef51c640fe366c4d6b1d799ed4eb2abab: Status 404 returned error can't find the container with id 96101031d4d65939894ad39fa928e44ef51c640fe366c4d6b1d799ed4eb2abab
Nov 03 07:22:22.690125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:22.690103    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:23.179158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:23.179041    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs" event={"ID":"3003102d-67e5-480d-8f93-401e01e2f017","Type":"ContainerStarted","Data":"af0eb1357cd182620c39180d3ae05561f42dd297fe3182db16e110863623dd71"}
Nov 03 07:22:23.179158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:23.179109    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs" event={"ID":"3003102d-67e5-480d-8f93-401e01e2f017","Type":"ContainerStarted","Data":"96101031d4d65939894ad39fa928e44ef51c640fe366c4d6b1d799ed4eb2abab"}
Nov 03 07:22:23.203159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:23.198968    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"]
Nov 03 07:22:23.396217 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:23.390933    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs" podStartSLOduration=2.390858376 podStartE2EDuration="2.390858376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:23.366554678 +0000 UTC m=+1395.698923673" watchObservedRunningTime="2026-11-03 07:22:23.390858376 +0000 UTC m=+1395.723227392"
Nov 03 07:22:24.539160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.420810    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:22:24.647823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.647668    4470 generic.go:356] "Generic (PLEG): container finished" podID="3003102d-67e5-480d-8f93-401e01e2f017" containerID="af0eb1357cd182620c39180d3ae05561f42dd297fe3182db16e110863623dd71" exitCode=0
Nov 03 07:22:24.665160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.648409    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs" event={"ID":"3003102d-67e5-480d-8f93-401e01e2f017","Type":"ContainerDied","Data":"af0eb1357cd182620c39180d3ae05561f42dd297fe3182db16e110863623dd71"}
Nov 03 07:22:24.667069 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.648466    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:24.730271 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.684527    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:22:24.743026 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.742240    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"]
Nov 03 07:22:24.768608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.767509    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:24.769718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.769036    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:24.839200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.838846    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:24.897160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.874691    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:24.897160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.880117    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:24.909775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.908455    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:24.977161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.960751    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:24.979310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:24.979282    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"]
Nov 03 07:22:25.033847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.033807    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q" event={"ID":"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad","Type":"ContainerStarted","Data":"22d9588ec67a6f55d1d111ce8b8d2ce90327f88798e0ea89324cd447ce023218"}
Nov 03 07:22:25.100987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.100892    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q" event={"ID":"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad","Type":"ContainerStarted","Data":"ab0c019bea33624c84dfcf1e7cf66f05897a5b6916301dfb04617bf1aa554c47"}
Nov 03 07:22:25.122288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.061348    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:22:25.123898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.062292    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.124179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.124149    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.148283 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.148242    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.148621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.148595    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.149015 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.148994    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.149292 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.149272    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.149503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.149485    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8545v\" (UniqueName: \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-kube-api-access-8545v\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.149694 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.149676    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.149848 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.149831    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.150075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.061297    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:25.215228 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.198794    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7141-5ffb7b"]
Nov 03 07:22:25.225200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.225122    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7141-5ffb7b"
Nov 03 07:22:25.281219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.268064    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.281219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.268118    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.281219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.268249    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.281219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.268293    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.281219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.268340    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.281219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.268406    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.281219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.268533    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.281219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.268593    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.281219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.268660    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8545v\" (UniqueName: \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-kube-api-access-8545v\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.297404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.295123    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mnr67\""
Nov 03 07:22:25.321161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.312219    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.341700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.341311    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.342245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.342190    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.356030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.355956    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.373342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.373306    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26cpc\" (UniqueName: \"kubernetes.io/projected/a3aa5b30-e794-4859-b1a0-a974e95ad313-kube-api-access-26cpc\") pod \"test-curl-7141-5ffb7b\" (UID: \"a3aa5b30-e794-4859-b1a0-a974e95ad313\") " pod="opendatahub/test-curl-7141-5ffb7b"
Nov 03 07:22:25.375721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.375693    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.424162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.398578    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.424423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.399233    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7141-5ffb7b"]
Nov 03 07:22:25.424578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.415939    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8545v\" (UniqueName: \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-kube-api-access-8545v\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.429160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.425815    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.429160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.426350    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.519162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.517989    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26cpc\" (UniqueName: \"kubernetes.io/projected/a3aa5b30-e794-4859-b1a0-a974e95ad313-kube-api-access-26cpc\") pod \"test-curl-7141-5ffb7b\" (UID: \"a3aa5b30-e794-4859-b1a0-a974e95ad313\") " pod="opendatahub/test-curl-7141-5ffb7b"
Nov 03 07:22:25.605260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.600958    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:25.638215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.631941    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26cpc\" (UniqueName: \"kubernetes.io/projected/a3aa5b30-e794-4859-b1a0-a974e95ad313-kube-api-access-26cpc\") pod \"test-curl-7141-5ffb7b\" (UID: \"a3aa5b30-e794-4859-b1a0-a974e95ad313\") " pod="opendatahub/test-curl-7141-5ffb7b"
Nov 03 07:22:25.682793 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.664924    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7141-5ffb7b"
Nov 03 07:22:25.731178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.638472    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:25.731178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.684666    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:22:25.731178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.684735    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:25.731178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.685847    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:22:25.731178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.714288    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:25.731178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.714769    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:22:25.814175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.769233    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 07:22:25.814175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.769430    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 03 07:22:25.814175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.769629    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:22:25.814175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.769690    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:22:25.929239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.866672    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:25.937439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.937398    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:25.962216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.962157    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q" podStartSLOduration=3.9621121 podStartE2EDuration="3.9621121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:25.340709131 +0000 UTC m=+1397.673078126" watchObservedRunningTime="2026-11-03 07:22:25.9621121 +0000 UTC m=+1398.294481095"
Nov 03 07:22:25.985390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:25.966223    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:26.000730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.000688    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:26.125022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.124938    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:26.161159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.132603    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:22:26.165309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.165274    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:22:26.205181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.205078    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:22:26.205487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.205466    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:22:26.205636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.205623    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:22:26.228160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.166746    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:26.264761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.264696    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:22:26.366718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.366674    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:26.523642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.523490    4470 generic.go:356] "Generic (PLEG): container finished" podID="61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad" containerID="22d9588ec67a6f55d1d111ce8b8d2ce90327f88798e0ea89324cd447ce023218" exitCode=0
Nov 03 07:22:26.605274 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.545107    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q" event={"ID":"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad","Type":"ContainerDied","Data":"22d9588ec67a6f55d1d111ce8b8d2ce90327f88798e0ea89324cd447ce023218"}
Nov 03 07:22:26.697172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.685516    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:26.702296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.702252    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:22:26.725061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.724239    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Nov 03 07:22:26.783933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.781220    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:26.783933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.781638    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:26.878073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.878017    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:26.954351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.954302    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:26.981159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:26.975189    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:27.041820 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:27.001407    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:27.134350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:27.134302    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:22:27.291019 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:27.267229    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:27.714164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:27.707213    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:22:27.714164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:27.708167    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:22:27.728571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:27.728521    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:27.900191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:27.892766    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:27.914197 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:27.896228    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 03 07:22:27.923159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:27.896284    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:27.941586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:27.941542    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:27.957936 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:27.957797    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:28.086211 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.063114    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 07:22:28.308410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.300638    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 07:22:28.323331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.323232    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:28.327341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.326045    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:28.391655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.391626    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:22:28.392007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.391990    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:22:28.392335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.392317    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:22:28.392527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.392513    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:22:28.392670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.392659    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:22:28.392813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.392802    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:22:28.462369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.452256    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:22:28.462369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.453016    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:22:28.462369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.453069    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:22:28.462369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.453103    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:22:28.462369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.456882    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:22:28.520748 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.520664    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:28.604487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.604415    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:28.655776 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.655746    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-kube-api-access-9d2xr\" (UniqueName: \"kubernetes.io/projected/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-kube-api-access-9d2xr\") pod \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") "
Nov 03 07:22:28.655954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.655826    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-openshift-ca\") pod \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") "
Nov 03 07:22:28.655954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.655874    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3003102d-67e5-480d-8f93-401e01e2f017-kube-api-access-t6rgf\" (UniqueName: \"kubernetes.io/projected/3003102d-67e5-480d-8f93-401e01e2f017-kube-api-access-t6rgf\") pod \"3003102d-67e5-480d-8f93-401e01e2f017\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") "
Nov 03 07:22:28.656071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.655989    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-xks-ca\") pod \"3003102d-67e5-480d-8f93-401e01e2f017\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") "
Nov 03 07:22:28.656071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.656039    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-openshift-ca\") pod \"3003102d-67e5-480d-8f93-401e01e2f017\" (UID: \"3003102d-67e5-480d-8f93-401e01e2f017\") "
Nov 03 07:22:28.656204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.656074    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-xks-ca\") pod \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\" (UID: \"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad\") "
Nov 03 07:22:28.656317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.656290    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-openshift-ca" pod "61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad" (UID: "61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:28.656452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.656357    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-openshift-ca" pod "3003102d-67e5-480d-8f93-401e01e2f017" (UID: "3003102d-67e5-480d-8f93-401e01e2f017"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:28.656768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.656745    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:28.656854 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.656774    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:28.658213 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.658108    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-kube-api-access-9d2xr" pod "61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad" (UID: "61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad"). InnerVolumeSpecName "kube-api-access-9d2xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:28.658213 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.658158    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-xks-ca" pod "3003102d-67e5-480d-8f93-401e01e2f017" (UID: "3003102d-67e5-480d-8f93-401e01e2f017"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:28.658796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.658770    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-xks-ca" pod "61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad" (UID: "61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:28.658796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.658791    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3003102d-67e5-480d-8f93-401e01e2f017-kube-api-access-t6rgf" pod "3003102d-67e5-480d-8f93-401e01e2f017" (UID: "3003102d-67e5-480d-8f93-401e01e2f017"). InnerVolumeSpecName "kube-api-access-t6rgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:28.675415 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:28.673559    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3aa5b30_e794_4859_b1a0_a974e95ad313.slice/crio-af3d0719578e58feda9e6591cb8beebcace27af9feabde98f4d145583c3793f1 WatchSource:0}: Error finding container af3d0719578e58feda9e6591cb8beebcace27af9feabde98f4d145583c3793f1: Status 404 returned error can't find the container with id af3d0719578e58feda9e6591cb8beebcace27af9feabde98f4d145583c3793f1
Nov 03 07:22:28.675604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.675586    4470 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 07:22:28.676316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.676290    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7141-5ffb7b"]
Nov 03 07:22:28.719650 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.719619    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"
Nov 03 07:22:28.719650 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.719637    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs" event={"ID":"3003102d-67e5-480d-8f93-401e01e2f017","Type":"ContainerDied","Data":"96101031d4d65939894ad39fa928e44ef51c640fe366c4d6b1d799ed4eb2abab"}
Nov 03 07:22:28.720178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.719702    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96101031d4d65939894ad39fa928e44ef51c640fe366c4d6b1d799ed4eb2abab"
Nov 03 07:22:28.721323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.721302    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"
Nov 03 07:22:28.721444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.721326    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q" event={"ID":"61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad","Type":"ContainerDied","Data":"ab0c019bea33624c84dfcf1e7cf66f05897a5b6916301dfb04617bf1aa554c47"}
Nov 03 07:22:28.721444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.721373    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab0c019bea33624c84dfcf1e7cf66f05897a5b6916301dfb04617bf1aa554c47"
Nov 03 07:22:28.722941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.722916    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7141-5ffb7b" event={"ID":"a3aa5b30-e794-4859-b1a0-a974e95ad313","Type":"ContainerStarted","Data":"af3d0719578e58feda9e6591cb8beebcace27af9feabde98f4d145583c3793f1"}
Nov 03 07:22:28.732294 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:28.732266    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284f7fe9_4542_47c4_bb83_ff3b25260bcd.slice/crio-1222b8e5074139b6c79d7796e1d642b71cd62c3e44a800ae53f0412a927ac392 WatchSource:0}: Error finding container 1222b8e5074139b6c79d7796e1d642b71cd62c3e44a800ae53f0412a927ac392: Status 404 returned error can't find the container with id 1222b8e5074139b6c79d7796e1d642b71cd62c3e44a800ae53f0412a927ac392
Nov 03 07:22:28.734491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.734372    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"]
Nov 03 07:22:28.735629 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.735591    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:28.735744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.735661    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:28.735744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.735692    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:28.758938 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.758908    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:28.759054 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.758948    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2xr\" (UniqueName: \"kubernetes.io/projected/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad-kube-api-access-9d2xr\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:28.759054 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.758966    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6rgf\" (UniqueName: \"kubernetes.io/projected/3003102d-67e5-480d-8f93-401e01e2f017-kube-api-access-t6rgf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:28.759054 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:28.758981    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3003102d-67e5-480d-8f93-401e01e2f017-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:29.181981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.181946    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"]
Nov 03 07:22:29.182608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.182590    4470 cpu_manager.go:414] "containerMap: removing container" podUID="61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad" containerName="revoke-keys"
Nov 03 07:22:29.182608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.182608    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad" containerName="revoke-keys"
Nov 03 07:22:29.182753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.182650    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3003102d-67e5-480d-8f93-401e01e2f017" containerName="revoke-keys"
Nov 03 07:22:29.182753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.182655    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3003102d-67e5-480d-8f93-401e01e2f017" containerName="revoke-keys"
Nov 03 07:22:29.182816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.182752    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad" containerName="revoke-keys"
Nov 03 07:22:29.182816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.182770    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3003102d-67e5-480d-8f93-401e01e2f017" containerName="revoke-keys"
Nov 03 07:22:29.186319 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.186305    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:29.249357 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.249316    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:22:29.249882 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.249856    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:22:29.250244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.250206    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:22:29.250480 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.250461    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:22:29.252738 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.252717    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"]
Nov 03 07:22:29.266424 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.266399    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9f39a0-v5kml\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:29.266530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.266456    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9f39a0-v5kml\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:29.266723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.266699    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8fk5\" (UniqueName: \"kubernetes.io/projected/ae19d29c-66cf-4d85-8500-8880f6aad534-kube-api-access-n8fk5\") pod \"maas-api-revoke-sub-9f9f39a0-v5kml\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:29.367710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.367678    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9f39a0-v5kml\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:29.367870 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.367827    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8fk5\" (UniqueName: \"kubernetes.io/projected/ae19d29c-66cf-4d85-8500-8880f6aad534-kube-api-access-n8fk5\") pod \"maas-api-revoke-sub-9f9f39a0-v5kml\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:29.367950 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.367935    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9f39a0-v5kml\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:29.368608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.368583    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9f39a0-v5kml\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:29.370338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.370318    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9f39a0-v5kml\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:29.440041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.439927    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8fk5\" (UniqueName: \"kubernetes.io/projected/ae19d29c-66cf-4d85-8500-8880f6aad534-kube-api-access-n8fk5\") pod \"maas-api-revoke-sub-9f9f39a0-v5kml\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:29.500394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.500368    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:29.555185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.549781    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"]
Nov 03 07:22:29.577962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.577638    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:29.616393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.616346    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"]
Nov 03 07:22:29.679006 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.678967    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51be60f0-vrqvh\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:29.679234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.679104    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51be60f0-vrqvh\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:29.679234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.679183    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw5qx\" (UniqueName: \"kubernetes.io/projected/88f03e02-0ff9-443c-91c2-845857d60d8b-kube-api-access-rw5qx\") pod \"maas-api-revoke-sub-51be60f0-vrqvh\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:29.743000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.742890    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" event={"ID":"284f7fe9-4542-47c4-bb83-ff3b25260bcd","Type":"ContainerStarted","Data":"ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26"}
Nov 03 07:22:29.743000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.742951    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" event={"ID":"284f7fe9-4542-47c4-bb83-ff3b25260bcd","Type":"ContainerStarted","Data":"1222b8e5074139b6c79d7796e1d642b71cd62c3e44a800ae53f0412a927ac392"}
Nov 03 07:22:29.838197 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.835062    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51be60f0-vrqvh\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:29.838197 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.835244    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51be60f0-vrqvh\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:29.838197 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.835284    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw5qx\" (UniqueName: \"kubernetes.io/projected/88f03e02-0ff9-443c-91c2-845857d60d8b-kube-api-access-rw5qx\") pod \"maas-api-revoke-sub-51be60f0-vrqvh\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:29.838197 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.836359    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51be60f0-vrqvh\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:29.884157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.880448    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51be60f0-vrqvh\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:29.922256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.921409    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw5qx\" (UniqueName: \"kubernetes.io/projected/88f03e02-0ff9-443c-91c2-845857d60d8b-kube-api-access-rw5qx\") pod \"maas-api-revoke-sub-51be60f0-vrqvh\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:29.997572 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:29.997548    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"]
Nov 03 07:22:30.203760 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:30.203729    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:30.602052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:30.602022    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:30.603711 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:30.603683    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 07:22:30.603832 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:30.603731    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 07:22:30.625253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:30.625225    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"]
Nov 03 07:22:30.625350 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:30.625298    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f03e02_0ff9_443c_91c2_845857d60d8b.slice/crio-0dd6beb70711ec42e51dc088f748bf31bd0e6d6a675b129d37d326940f42f54e WatchSource:0}: Error finding container 0dd6beb70711ec42e51dc088f748bf31bd0e6d6a675b129d37d326940f42f54e: Status 404 returned error can't find the container with id 0dd6beb70711ec42e51dc088f748bf31bd0e6d6a675b129d37d326940f42f54e
Nov 03 07:22:30.778739 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:30.778705    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh" event={"ID":"88f03e02-0ff9-443c-91c2-845857d60d8b","Type":"ContainerStarted","Data":"0dd6beb70711ec42e51dc088f748bf31bd0e6d6a675b129d37d326940f42f54e"}
Nov 03 07:22:30.781309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:30.781283    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml" event={"ID":"ae19d29c-66cf-4d85-8500-8880f6aad534","Type":"ContainerStarted","Data":"d051da0d5cc72394c239a14023c574f31c6fb41876c6c26ab6151ad913d06a60"}
Nov 03 07:22:30.781422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:30.781331    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml" event={"ID":"ae19d29c-66cf-4d85-8500-8880f6aad534","Type":"ContainerStarted","Data":"04d93ce8bf19b69ee67353be6eecc69db10e117cc2718d904ed4df751ac0ce86"}
Nov 03 07:22:30.980923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:30.980869    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml" podStartSLOduration=1.980854521 podStartE2EDuration="1.980854521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:30.976939575 +0000 UTC m=+1403.309308568" watchObservedRunningTime="2026-11-03 07:22:30.980854521 +0000 UTC m=+1403.313223515"
Nov 03 07:22:31.105190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:31.105117    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" podStartSLOduration=7.10509855 podStartE2EDuration="7.10509855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:31.103587996 +0000 UTC m=+1403.435956991" watchObservedRunningTime="2026-11-03 07:22:31.10509855 +0000 UTC m=+1403.437467545"
Nov 03 07:22:31.601789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:31.601749    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 07:22:31.601968 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:31.601809    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 07:22:31.791124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:31.791090    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh" event={"ID":"88f03e02-0ff9-443c-91c2-845857d60d8b","Type":"ContainerStarted","Data":"9ebfdf73c522a5c887d4ae6c66c31692bc3ab0a7f2e21de29288bac68704191a"}
Nov 03 07:22:31.793076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:31.793046    4470 generic.go:356] "Generic (PLEG): container finished" podID="ae19d29c-66cf-4d85-8500-8880f6aad534" containerID="d051da0d5cc72394c239a14023c574f31c6fb41876c6c26ab6151ad913d06a60" exitCode=0
Nov 03 07:22:31.793285 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:31.793243    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml" event={"ID":"ae19d29c-66cf-4d85-8500-8880f6aad534","Type":"ContainerDied","Data":"d051da0d5cc72394c239a14023c574f31c6fb41876c6c26ab6151ad913d06a60"}
Nov 03 07:22:31.918199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:31.916908    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh" podStartSLOduration=2.916889314 podStartE2EDuration="2.916889314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:31.913822396 +0000 UTC m=+1404.246191407" watchObservedRunningTime="2026-11-03 07:22:31.916889314 +0000 UTC m=+1404.249258312"
Nov 03 07:22:32.601729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.601698    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 07:22:32.601865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.601747    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 07:22:32.701690 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.701660    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"]
Nov 03 07:22:32.706030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.706012    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:32.778428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.778341    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"]
Nov 03 07:22:32.803102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.803069    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a6a6d68-f6fcr\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:32.803541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.803150    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a6a6d68-f6fcr\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:32.803541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.803337    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktng\" (UniqueName: \"kubernetes.io/projected/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-kube-api-access-sktng\") pod \"maas-api-revoke-sub-0a6a6d68-f6fcr\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:32.806261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.806231    4470 generic.go:356] "Generic (PLEG): container finished" podID="88f03e02-0ff9-443c-91c2-845857d60d8b" containerID="9ebfdf73c522a5c887d4ae6c66c31692bc3ab0a7f2e21de29288bac68704191a" exitCode=0
Nov 03 07:22:32.806440 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.806405    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh" event={"ID":"88f03e02-0ff9-443c-91c2-845857d60d8b","Type":"ContainerDied","Data":"9ebfdf73c522a5c887d4ae6c66c31692bc3ab0a7f2e21de29288bac68704191a"}
Nov 03 07:22:32.808551 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.808526    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7141-5ffb7b" event={"ID":"a3aa5b30-e794-4859-b1a0-a974e95ad313","Type":"ContainerStarted","Data":"4e6e03c375a81b343c2511fbf334f82a9da296cd21a1589f24faa72ea958fc1d"}
Nov 03 07:22:32.826114 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.826090    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"]
Nov 03 07:22:32.830703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.830679    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:32.883805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.883768    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"]
Nov 03 07:22:32.905019 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.904988    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a6a6d68-f6fcr\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:32.905175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.905063    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a6a6d68-f6fcr\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:32.905875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.905848    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktng\" (UniqueName: \"kubernetes.io/projected/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-kube-api-access-sktng\") pod \"maas-api-revoke-sub-0a6a6d68-f6fcr\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:32.906089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.906071    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a6a6d68-f6fcr\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:32.910048 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.908270    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a6a6d68-f6fcr\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:32.980998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:32.980961    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktng\" (UniqueName: \"kubernetes.io/projected/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-kube-api-access-sktng\") pod \"maas-api-revoke-sub-0a6a6d68-f6fcr\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:33.007592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.007560    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ab3f37-g44w9\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:33.007767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.007653    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ab3f37-g44w9\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:33.007767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.007677    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl2dn\" (UniqueName: \"kubernetes.io/projected/c4534034-0f85-4eaf-8fc6-11c8c130c739-kube-api-access-bl2dn\") pod \"maas-api-revoke-sub-96ab3f37-g44w9\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:33.034241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.034170    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:33.037223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.037205    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:33.112260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.111966    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ab3f37-g44w9\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:33.112260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.112082    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ab3f37-g44w9\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:33.112260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.112127    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl2dn\" (UniqueName: \"kubernetes.io/projected/c4534034-0f85-4eaf-8fc6-11c8c130c739-kube-api-access-bl2dn\") pod \"maas-api-revoke-sub-96ab3f37-g44w9\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:33.113663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.113640    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ab3f37-g44w9\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:33.115579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.115556    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ab3f37-g44w9\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:33.178106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.178074    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl2dn\" (UniqueName: \"kubernetes.io/projected/c4534034-0f85-4eaf-8fc6-11c8c130c739-kube-api-access-bl2dn\") pod \"maas-api-revoke-sub-96ab3f37-g44w9\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:33.212012 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.211953    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7141-5ffb7b" podStartSLOduration=5.357768773 podStartE2EDuration="9.211938864s" totalImagesPullingTime="3.854170091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:33.207192542 +0000 UTC m=+1405.539561537" watchObservedRunningTime="2026-11-03 07:22:33.211938864 +0000 UTC m=+1405.544307858"
Nov 03 07:22:33.213616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.213592    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-openshift-ca\") pod \"ae19d29c-66cf-4d85-8500-8880f6aad534\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") "
Nov 03 07:22:33.213744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.213716    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae19d29c-66cf-4d85-8500-8880f6aad534-kube-api-access-n8fk5\" (UniqueName: \"kubernetes.io/projected/ae19d29c-66cf-4d85-8500-8880f6aad534-kube-api-access-n8fk5\") pod \"ae19d29c-66cf-4d85-8500-8880f6aad534\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") "
Nov 03 07:22:33.213813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.213742    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-xks-ca\") pod \"ae19d29c-66cf-4d85-8500-8880f6aad534\" (UID: \"ae19d29c-66cf-4d85-8500-8880f6aad534\") "
Nov 03 07:22:33.214055 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.214024    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-openshift-ca" pod "ae19d29c-66cf-4d85-8500-8880f6aad534" (UID: "ae19d29c-66cf-4d85-8500-8880f6aad534"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:33.214370 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.214349    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:33.215917 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.215899    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-xks-ca" pod "ae19d29c-66cf-4d85-8500-8880f6aad534" (UID: "ae19d29c-66cf-4d85-8500-8880f6aad534"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:33.215965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.215944    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae19d29c-66cf-4d85-8500-8880f6aad534-kube-api-access-n8fk5" pod "ae19d29c-66cf-4d85-8500-8880f6aad534" (UID: "ae19d29c-66cf-4d85-8500-8880f6aad534"). InnerVolumeSpecName "kube-api-access-n8fk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:33.315836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.315807    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8fk5\" (UniqueName: \"kubernetes.io/projected/ae19d29c-66cf-4d85-8500-8880f6aad534-kube-api-access-n8fk5\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:33.316031 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.316011    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae19d29c-66cf-4d85-8500-8880f6aad534-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:33.439552 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:33.439521    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c897f4_a4ac_4d35_8ce7_1a7d82ff4fe8.slice/crio-20f1a506a7d4ab797acd8a21b94dab1b5b87b854f3a8cadc69d36867b1211a41 WatchSource:0}: Error finding container 20f1a506a7d4ab797acd8a21b94dab1b5b87b854f3a8cadc69d36867b1211a41: Status 404 returned error can't find the container with id 20f1a506a7d4ab797acd8a21b94dab1b5b87b854f3a8cadc69d36867b1211a41
Nov 03 07:22:33.440009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.439986    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"]
Nov 03 07:22:33.474031 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.473998    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:33.601926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.601836    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 07:22:33.601926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.601896    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 07:22:33.818863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.818833    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr" event={"ID":"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8","Type":"ContainerStarted","Data":"20f1a506a7d4ab797acd8a21b94dab1b5b87b854f3a8cadc69d36867b1211a41"}
Nov 03 07:22:33.820185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.820167    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"
Nov 03 07:22:33.820305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.820166    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml" event={"ID":"ae19d29c-66cf-4d85-8500-8880f6aad534","Type":"ContainerDied","Data":"04d93ce8bf19b69ee67353be6eecc69db10e117cc2718d904ed4df751ac0ce86"}
Nov 03 07:22:33.820305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.820285    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d93ce8bf19b69ee67353be6eecc69db10e117cc2718d904ed4df751ac0ce86"
Nov 03 07:22:33.879107 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:33.879073    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4534034_0f85_4eaf_8fc6_11c8c130c739.slice/crio-e30aacb5ea868856b885d5a90913e5d57e49cb3aa0df3e1444f7a185a0caf7a8 WatchSource:0}: Error finding container e30aacb5ea868856b885d5a90913e5d57e49cb3aa0df3e1444f7a185a0caf7a8: Status 404 returned error can't find the container with id e30aacb5ea868856b885d5a90913e5d57e49cb3aa0df3e1444f7a185a0caf7a8
Nov 03 07:22:33.880998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:33.880924    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"]
Nov 03 07:22:34.022835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.022809    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:34.128782 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.128707    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f03e02-0ff9-443c-91c2-845857d60d8b-kube-api-access-rw5qx\" (UniqueName: \"kubernetes.io/projected/88f03e02-0ff9-443c-91c2-845857d60d8b-kube-api-access-rw5qx\") pod \"88f03e02-0ff9-443c-91c2-845857d60d8b\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") "
Nov 03 07:22:34.128782 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.128773    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-xks-ca\") pod \"88f03e02-0ff9-443c-91c2-845857d60d8b\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") "
Nov 03 07:22:34.128962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.128874    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-openshift-ca\") pod \"88f03e02-0ff9-443c-91c2-845857d60d8b\" (UID: \"88f03e02-0ff9-443c-91c2-845857d60d8b\") "
Nov 03 07:22:34.129265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.129237    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-openshift-ca" pod "88f03e02-0ff9-443c-91c2-845857d60d8b" (UID: "88f03e02-0ff9-443c-91c2-845857d60d8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:34.129512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.129493    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:34.130966 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.130938    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f03e02-0ff9-443c-91c2-845857d60d8b-kube-api-access-rw5qx" pod "88f03e02-0ff9-443c-91c2-845857d60d8b" (UID: "88f03e02-0ff9-443c-91c2-845857d60d8b"). InnerVolumeSpecName "kube-api-access-rw5qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:34.131078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.130978    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-xks-ca" pod "88f03e02-0ff9-443c-91c2-845857d60d8b" (UID: "88f03e02-0ff9-443c-91c2-845857d60d8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:34.231094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.231063    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw5qx\" (UniqueName: \"kubernetes.io/projected/88f03e02-0ff9-443c-91c2-845857d60d8b-kube-api-access-rw5qx\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:34.231094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.231090    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88f03e02-0ff9-443c-91c2-845857d60d8b-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:34.602405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.602366    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 07:22:34.602572 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.602439    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 07:22:34.780232 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:22:34.780190    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:22:34.802624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.802543    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7141-5ffb7b"]
Nov 03 07:22:34.810489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.810459    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7141-5ffb7b"]
Nov 03 07:22:34.831204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.831098    4470 generic.go:356] "Generic (PLEG): container finished" podID="00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8" containerID="f563e55bfca05660c54a59a8d2d0079e842eb5df28366212067d453f79e829bf" exitCode=0
Nov 03 07:22:34.831204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.831167    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr" event={"ID":"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8","Type":"ContainerDied","Data":"f563e55bfca05660c54a59a8d2d0079e842eb5df28366212067d453f79e829bf"}
Nov 03 07:22:34.832770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.832747    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh" event={"ID":"88f03e02-0ff9-443c-91c2-845857d60d8b","Type":"ContainerDied","Data":"0dd6beb70711ec42e51dc088f748bf31bd0e6d6a675b129d37d326940f42f54e"}
Nov 03 07:22:34.832883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.832784    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"
Nov 03 07:22:34.833329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.832790    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dd6beb70711ec42e51dc088f748bf31bd0e6d6a675b129d37d326940f42f54e"
Nov 03 07:22:34.839350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.839318    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7141-5ffb7b" podUID="a3aa5b30-e794-4859-b1a0-a974e95ad313" containerName="test-curl-7141-5ffb7b" containerID="cri-o://4e6e03c375a81b343c2511fbf334f82a9da296cd21a1589f24faa72ea958fc1d" gracePeriod=2
Nov 03 07:22:34.839757 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.839616    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9" event={"ID":"c4534034-0f85-4eaf-8fc6-11c8c130c739","Type":"ContainerStarted","Data":"b3fe78465965ee147ee8d9beb8a86360063e069288e6894cd33245bb506826f6"}
Nov 03 07:22:34.839757 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.839652    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9" event={"ID":"c4534034-0f85-4eaf-8fc6-11c8c130c739","Type":"ContainerStarted","Data":"e30aacb5ea868856b885d5a90913e5d57e49cb3aa0df3e1444f7a185a0caf7a8"}
Nov 03 07:22:34.977671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.977641    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7141-e03248"]
Nov 03 07:22:34.978268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.978254    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ae19d29c-66cf-4d85-8500-8880f6aad534" containerName="revoke-keys"
Nov 03 07:22:34.978318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.978270    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae19d29c-66cf-4d85-8500-8880f6aad534" containerName="revoke-keys"
Nov 03 07:22:34.978318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.978295    4470 cpu_manager.go:414] "containerMap: removing container" podUID="a3aa5b30-e794-4859-b1a0-a974e95ad313" containerName="test-curl-7141-5ffb7b"
Nov 03 07:22:34.978318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.978300    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3aa5b30-e794-4859-b1a0-a974e95ad313" containerName="test-curl-7141-5ffb7b"
Nov 03 07:22:34.978412 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.978327    4470 cpu_manager.go:414] "containerMap: removing container" podUID="88f03e02-0ff9-443c-91c2-845857d60d8b" containerName="revoke-keys"
Nov 03 07:22:34.978412 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.978332    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f03e02-0ff9-443c-91c2-845857d60d8b" containerName="revoke-keys"
Nov 03 07:22:34.978472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.978420    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f03e02-0ff9-443c-91c2-845857d60d8b" containerName="revoke-keys"
Nov 03 07:22:34.978472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.978432    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae19d29c-66cf-4d85-8500-8880f6aad534" containerName="revoke-keys"
Nov 03 07:22:34.978472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.978441    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3aa5b30-e794-4859-b1a0-a974e95ad313" containerName="test-curl-7141-5ffb7b"
Nov 03 07:22:34.982201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:34.982183    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7141-e03248"
Nov 03 07:22:35.059516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.059467    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"]
Nov 03 07:22:35.065341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.065310    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5876a62a-nmg9n"]
Nov 03 07:22:35.070842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.070812    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7141-e03248"]
Nov 03 07:22:35.132900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.132789    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9" podStartSLOduration=3.132775422 podStartE2EDuration="3.132775422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:35.1298148 +0000 UTC m=+1407.462183806" watchObservedRunningTime="2026-11-03 07:22:35.132775422 +0000 UTC m=+1407.465144419"
Nov 03 07:22:35.143692 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.143662    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzlfd\" (UniqueName: \"kubernetes.io/projected/aa4b4f35-8784-4b3c-9f8f-e29409a82928-kube-api-access-fzlfd\") pod \"test-curl-7141-e03248\" (UID: \"aa4b4f35-8784-4b3c-9f8f-e29409a82928\") " pod="opendatahub/test-curl-7141-e03248"
Nov 03 07:22:35.182628 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:22:35.182592    4470 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7141-5ffb7b\" is forbidden: User \"system:node:ip-10-0-129-24.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-24.us-west-1.compute.internal' and this object" podUID="a3aa5b30-e794-4859-b1a0-a974e95ad313" pod="opendatahub/test-curl-7141-5ffb7b"
Nov 03 07:22:35.245259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.245210    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlfd\" (UniqueName: \"kubernetes.io/projected/aa4b4f35-8784-4b3c-9f8f-e29409a82928-kube-api-access-fzlfd\") pod \"test-curl-7141-e03248\" (UID: \"aa4b4f35-8784-4b3c-9f8f-e29409a82928\") " pod="opendatahub/test-curl-7141-e03248"
Nov 03 07:22:35.311598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.311570    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlfd\" (UniqueName: \"kubernetes.io/projected/aa4b4f35-8784-4b3c-9f8f-e29409a82928-kube-api-access-fzlfd\") pod \"test-curl-7141-e03248\" (UID: \"aa4b4f35-8784-4b3c-9f8f-e29409a82928\") " pod="opendatahub/test-curl-7141-e03248"
Nov 03 07:22:35.353322 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:22:35.353289    4470 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7141-5ffb7b\" is forbidden: User \"system:node:ip-10-0-129-24.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-24.us-west-1.compute.internal' and this object" podUID="a3aa5b30-e794-4859-b1a0-a974e95ad313" pod="opendatahub/test-curl-7141-5ffb7b"
Nov 03 07:22:35.596243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.596207    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7141-e03248"
Nov 03 07:22:35.606527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.606494    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 07:22:35.606692 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.606574    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 07:22:35.808082 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.808050    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22d4004-93ee-459c-a354-cce7ac13680e" path="/var/lib/kubelet/pods/c22d4004-93ee-459c-a354-cce7ac13680e/volumes"
Nov 03 07:22:35.853308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.853209    4470 generic.go:356] "Generic (PLEG): container finished" podID="c4534034-0f85-4eaf-8fc6-11c8c130c739" containerID="b3fe78465965ee147ee8d9beb8a86360063e069288e6894cd33245bb506826f6" exitCode=0
Nov 03 07:22:35.853308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.853256    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9" event={"ID":"c4534034-0f85-4eaf-8fc6-11c8c130c739","Type":"ContainerDied","Data":"b3fe78465965ee147ee8d9beb8a86360063e069288e6894cd33245bb506826f6"}
Nov 03 07:22:35.992748 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:35.992712    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa4b4f35_8784_4b3c_9f8f_e29409a82928.slice/crio-1f805da8cbe1f163c6b46aff615f97efd880f8051b40bc7e3ff0b67f29868e5b WatchSource:0}: Error finding container 1f805da8cbe1f163c6b46aff615f97efd880f8051b40bc7e3ff0b67f29868e5b: Status 404 returned error can't find the container with id 1f805da8cbe1f163c6b46aff615f97efd880f8051b40bc7e3ff0b67f29868e5b
Nov 03 07:22:35.996195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:35.996173    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7141-e03248"]
Nov 03 07:22:36.076600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.076575    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:36.157367 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.157257    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-xks-ca\") pod \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") "
Nov 03 07:22:36.157539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.157472    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-openshift-ca\") pod \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") "
Nov 03 07:22:36.157616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.157541    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-kube-api-access-sktng\" (UniqueName: \"kubernetes.io/projected/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-kube-api-access-sktng\") pod \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\" (UID: \"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8\") "
Nov 03 07:22:36.157817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.157794    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-openshift-ca" pod "00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8" (UID: "00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:36.158244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.158223    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:36.159571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.159536    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-xks-ca" pod "00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8" (UID: "00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:36.160202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.160181    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-kube-api-access-sktng" pod "00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8" (UID: "00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8"). InnerVolumeSpecName "kube-api-access-sktng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:36.260279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.260223    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sktng\" (UniqueName: \"kubernetes.io/projected/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-kube-api-access-sktng\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:36.260279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.260277    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:36.606098 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.606070    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:36.606566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.606550    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:36.607414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.607394    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:22:36.883696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.883656    4470 generic.go:356] "Generic (PLEG): container finished" podID="a3aa5b30-e794-4859-b1a0-a974e95ad313" containerID="4e6e03c375a81b343c2511fbf334f82a9da296cd21a1589f24faa72ea958fc1d" exitCode=137
Nov 03 07:22:36.892656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.892618    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7141-e03248" event={"ID":"aa4b4f35-8784-4b3c-9f8f-e29409a82928","Type":"ContainerStarted","Data":"e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636"}
Nov 03 07:22:36.894583 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.894558    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7141-e03248" event={"ID":"aa4b4f35-8784-4b3c-9f8f-e29409a82928","Type":"ContainerStarted","Data":"1f805da8cbe1f163c6b46aff615f97efd880f8051b40bc7e3ff0b67f29868e5b"}
Nov 03 07:22:36.895619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.895588    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"
Nov 03 07:22:36.897306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.897282    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr" event={"ID":"00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8","Type":"ContainerDied","Data":"20f1a506a7d4ab797acd8a21b94dab1b5b87b854f3a8cadc69d36867b1211a41"}
Nov 03 07:22:36.897477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:36.897462    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20f1a506a7d4ab797acd8a21b94dab1b5b87b854f3a8cadc69d36867b1211a41"
Nov 03 07:22:37.015024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.014967    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7141-e03248" podStartSLOduration=3.014952729 podStartE2EDuration="3.014952729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:37.010984661 +0000 UTC m=+1409.343353655" watchObservedRunningTime="2026-11-03 07:22:37.014952729 +0000 UTC m=+1409.347321724"
Nov 03 07:22:37.076706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.076680    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:37.112336 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.112282    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7141-5ffb7b"
Nov 03 07:22:37.178545 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.178514    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-openshift-ca\") pod \"c4534034-0f85-4eaf-8fc6-11c8c130c739\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") "
Nov 03 07:22:37.178705 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.178569    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-xks-ca\") pod \"c4534034-0f85-4eaf-8fc6-11c8c130c739\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") "
Nov 03 07:22:37.178705 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.178655    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4534034-0f85-4eaf-8fc6-11c8c130c739-kube-api-access-bl2dn\" (UniqueName: \"kubernetes.io/projected/c4534034-0f85-4eaf-8fc6-11c8c130c739-kube-api-access-bl2dn\") pod \"c4534034-0f85-4eaf-8fc6-11c8c130c739\" (UID: \"c4534034-0f85-4eaf-8fc6-11c8c130c739\") "
Nov 03 07:22:37.178817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.178757    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3aa5b30-e794-4859-b1a0-a974e95ad313-kube-api-access-26cpc\" (UniqueName: \"kubernetes.io/projected/a3aa5b30-e794-4859-b1a0-a974e95ad313-kube-api-access-26cpc\") pod \"a3aa5b30-e794-4859-b1a0-a974e95ad313\" (UID: \"a3aa5b30-e794-4859-b1a0-a974e95ad313\") "
Nov 03 07:22:37.178928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.178904    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-openshift-ca" pod "c4534034-0f85-4eaf-8fc6-11c8c130c739" (UID: "c4534034-0f85-4eaf-8fc6-11c8c130c739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:37.179321 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.179301    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:37.180926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.180901    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3aa5b30-e794-4859-b1a0-a974e95ad313-kube-api-access-26cpc" pod "a3aa5b30-e794-4859-b1a0-a974e95ad313" (UID: "a3aa5b30-e794-4859-b1a0-a974e95ad313"). InnerVolumeSpecName "kube-api-access-26cpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:37.181010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.180942    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4534034-0f85-4eaf-8fc6-11c8c130c739-kube-api-access-bl2dn" pod "c4534034-0f85-4eaf-8fc6-11c8c130c739" (UID: "c4534034-0f85-4eaf-8fc6-11c8c130c739"). InnerVolumeSpecName "kube-api-access-bl2dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:37.181175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.181145    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-xks-ca" pod "c4534034-0f85-4eaf-8fc6-11c8c130c739" (UID: "c4534034-0f85-4eaf-8fc6-11c8c130c739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:37.280313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.280278    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4534034-0f85-4eaf-8fc6-11c8c130c739-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:37.280313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.280306    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl2dn\" (UniqueName: \"kubernetes.io/projected/c4534034-0f85-4eaf-8fc6-11c8c130c739-kube-api-access-bl2dn\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:37.280313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.280317    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26cpc\" (UniqueName: \"kubernetes.io/projected/a3aa5b30-e794-4859-b1a0-a974e95ad313-kube-api-access-26cpc\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:37.809094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.809063    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3aa5b30-e794-4859-b1a0-a974e95ad313" path="/var/lib/kubelet/pods/a3aa5b30-e794-4859-b1a0-a974e95ad313/volumes"
Nov 03 07:22:37.904474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.904444    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"
Nov 03 07:22:37.904899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.904470    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9" event={"ID":"c4534034-0f85-4eaf-8fc6-11c8c130c739","Type":"ContainerDied","Data":"e30aacb5ea868856b885d5a90913e5d57e49cb3aa0df3e1444f7a185a0caf7a8"}
Nov 03 07:22:37.904899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.904522    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e30aacb5ea868856b885d5a90913e5d57e49cb3aa0df3e1444f7a185a0caf7a8"
Nov 03 07:22:37.906022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.906001    4470 scope.go:122] "RemoveContainer" containerID="4e6e03c375a81b343c2511fbf334f82a9da296cd21a1589f24faa72ea958fc1d"
Nov 03 07:22:37.906173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:37.906101    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7141-5ffb7b"
Nov 03 07:22:38.669983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:38.669952    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7141-e03248"]
Nov 03 07:22:38.678017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:38.677988    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7141-e03248"]
Nov 03 07:22:38.918206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:38.918154    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7141-e03248" podUID="aa4b4f35-8784-4b3c-9f8f-e29409a82928" containerName="test-curl-7141-e03248" containerID="cri-o://e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636" gracePeriod=2
Nov 03 07:22:38.960206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:38.957619    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"]
Nov 03 07:22:38.985156 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:22:38.982441    4470 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7141-e03248\" is forbidden: User \"system:node:ip-10-0-129-24.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-24.us-west-1.compute.internal' and this object" podUID="aa4b4f35-8784-4b3c-9f8f-e29409a82928" pod="opendatahub/test-curl-7141-e03248"
Nov 03 07:22:39.007599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.007566    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c4534034-0f85-4eaf-8fc6-11c8c130c739" containerName="revoke-keys"
Nov 03 07:22:39.007599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.007596    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4534034-0f85-4eaf-8fc6-11c8c130c739" containerName="revoke-keys"
Nov 03 07:22:39.007912 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.007645    4470 cpu_manager.go:414] "containerMap: removing container" podUID="aa4b4f35-8784-4b3c-9f8f-e29409a82928" containerName="test-curl-7141-e03248"
Nov 03 07:22:39.007912 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.007654    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4b4f35-8784-4b3c-9f8f-e29409a82928" containerName="test-curl-7141-e03248"
Nov 03 07:22:39.007912 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.007683    4470 cpu_manager.go:414] "containerMap: removing container" podUID="00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8" containerName="revoke-keys"
Nov 03 07:22:39.007912 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.007691    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8" containerName="revoke-keys"
Nov 03 07:22:39.007912 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.007855    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4b4f35-8784-4b3c-9f8f-e29409a82928" containerName="test-curl-7141-e03248"
Nov 03 07:22:39.007912 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.007872    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8" containerName="revoke-keys"
Nov 03 07:22:39.007912 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.007887    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4534034-0f85-4eaf-8fc6-11c8c130c739" containerName="revoke-keys"
Nov 03 07:22:39.055193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.050967    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:39.076647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.076617    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"]
Nov 03 07:22:39.106043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.105999    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:22:39.106227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.106086    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:22:39.107181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.107159    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:22:39.107694 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.107672    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9f69e6-9g4wd\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:39.107792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.107734    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9f69e6-9g4wd\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:39.107894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.107875    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6595\" (UniqueName: \"kubernetes.io/projected/8ea16936-8a96-44c1-a776-27c9c7cd6d70-kube-api-access-h6595\") pod \"maas-api-revoke-sub-9e9f69e6-9g4wd\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:39.156896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.156869    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:22:39.209163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.209104    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9f69e6-9g4wd\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:39.209347 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.209203    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9f69e6-9g4wd\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:39.209347 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.209253    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6595\" (UniqueName: \"kubernetes.io/projected/8ea16936-8a96-44c1-a776-27c9c7cd6d70-kube-api-access-h6595\") pod \"maas-api-revoke-sub-9e9f69e6-9g4wd\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:39.209786 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.209762    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9f69e6-9g4wd\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:39.211764 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.211719    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9f69e6-9g4wd\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:39.221949 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:22:39.221921    4470 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7141-e03248\" is forbidden: User \"system:node:ip-10-0-129-24.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-24.us-west-1.compute.internal' and this object" podUID="aa4b4f35-8784-4b3c-9f8f-e29409a82928" pod="opendatahub/test-curl-7141-e03248"
Nov 03 07:22:39.273805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.273771    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6595\" (UniqueName: \"kubernetes.io/projected/8ea16936-8a96-44c1-a776-27c9c7cd6d70-kube-api-access-h6595\") pod \"maas-api-revoke-sub-9e9f69e6-9g4wd\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:39.372778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.372744    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:39.380204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.380179    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"]
Nov 03 07:22:39.397870 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.397837    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.486875 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:22:39.450594    4470 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7141-e03248\" is forbidden: User \"system:node:ip-10-0-129-24.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-24.us-west-1.compute.internal' and this object" podUID="aa4b4f35-8784-4b3c-9f8f-e29409a82928" pod="opendatahub/test-curl-7141-e03248"
Nov 03 07:22:39.486875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.469586    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"]
Nov 03 07:22:39.513385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.513347    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.513603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.513454    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.513603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.513494    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.513778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.513620    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzt8j\" (UniqueName: \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-kube-api-access-vzt8j\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.513778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.513683    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.513778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.513768    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.514010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.513900    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.514010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.513934    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.514010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.513977    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.615213 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.615172    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.615398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.615276    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.615398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.615316    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.615527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.615411    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzt8j\" (UniqueName: \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-kube-api-access-vzt8j\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.615527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.615444    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.615527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.615511    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.615682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.615603    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.615682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.615648    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.615806 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.615693    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.616052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.615999    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.616052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.616034    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.616233 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.616086    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.616233 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.616128    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.616233 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.616151    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.617852 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.617824    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.618495 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.618472    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.695921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.695889    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.696055 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.696027    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzt8j\" (UniqueName: \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-kube-api-access-vzt8j\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.736217 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.736184    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:39.777685 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:39.777659    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea16936_8a96_44c1_a776_27c9c7cd6d70.slice/crio-5de09ed29a4127198ebfeea3b51571b39029cebb665135a751b14745d6b481b6 WatchSource:0}: Error finding container 5de09ed29a4127198ebfeea3b51571b39029cebb665135a751b14745d6b481b6: Status 404 returned error can't find the container with id 5de09ed29a4127198ebfeea3b51571b39029cebb665135a751b14745d6b481b6
Nov 03 07:22:39.779500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.779474    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"]
Nov 03 07:22:39.928391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:39.928359    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd" event={"ID":"8ea16936-8a96-44c1-a776-27c9c7cd6d70","Type":"ContainerStarted","Data":"5de09ed29a4127198ebfeea3b51571b39029cebb665135a751b14745d6b481b6"}
Nov 03 07:22:40.129481 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:40.129453    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79cad3f3_671b_47dd_8cb9_98d9e4b2439b.slice/crio-e6db7cc96f365fd86c94d061dda773e34c55890c00ab05b4c7fe9c27ada0f8b5 WatchSource:0}: Error finding container e6db7cc96f365fd86c94d061dda773e34c55890c00ab05b4c7fe9c27ada0f8b5: Status 404 returned error can't find the container with id e6db7cc96f365fd86c94d061dda773e34c55890c00ab05b4c7fe9c27ada0f8b5
Nov 03 07:22:40.131375 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.131354    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"]
Nov 03 07:22:40.131962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.131933    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:40.132024 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.132012    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:40.132067 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.132054    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:40.366476 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.366442    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"]
Nov 03 07:22:40.370937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.370914    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:40.427431 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.427398    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjkq\" (UniqueName: \"kubernetes.io/projected/e7e9755c-f3e3-4eae-ab90-fd646155683f-kube-api-access-vmjkq\") pod \"maas-api-revoke-sub-6ce89b84-tmpdt\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:40.427580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.427547    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce89b84-tmpdt\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:40.427651 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.427594    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce89b84-tmpdt\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:40.443589 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.443555    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"]
Nov 03 07:22:40.585160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.553101    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjkq\" (UniqueName: \"kubernetes.io/projected/e7e9755c-f3e3-4eae-ab90-fd646155683f-kube-api-access-vmjkq\") pod \"maas-api-revoke-sub-6ce89b84-tmpdt\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:40.608726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.595438    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce89b84-tmpdt\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:40.608726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.595571    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce89b84-tmpdt\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:40.608726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.599253    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce89b84-tmpdt\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:40.608726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.599971    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce89b84-tmpdt\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:40.649688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.649619    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjkq\" (UniqueName: \"kubernetes.io/projected/e7e9755c-f3e3-4eae-ab90-fd646155683f-kube-api-access-vmjkq\") pod \"maas-api-revoke-sub-6ce89b84-tmpdt\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:40.688037 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.688004    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:40.941882 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.941851    4470 generic.go:356] "Generic (PLEG): container finished" podID="8ea16936-8a96-44c1-a776-27c9c7cd6d70" containerID="cfc317cc5a93d173ba3f8232259a6aad09ebcabc697803ce82902cdfdf8ddfbc" exitCode=0
Nov 03 07:22:40.942308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.941946    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd" event={"ID":"8ea16936-8a96-44c1-a776-27c9c7cd6d70","Type":"ContainerDied","Data":"cfc317cc5a93d173ba3f8232259a6aad09ebcabc697803ce82902cdfdf8ddfbc"}
Nov 03 07:22:40.943895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.943872    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" event={"ID":"79cad3f3-671b-47dd-8cb9-98d9e4b2439b","Type":"ContainerStarted","Data":"a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2"}
Nov 03 07:22:40.943997 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:40.943920    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" event={"ID":"79cad3f3-671b-47dd-8cb9-98d9e4b2439b","Type":"ContainerStarted","Data":"e6db7cc96f365fd86c94d061dda773e34c55890c00ab05b4c7fe9c27ada0f8b5"}
Nov 03 07:22:41.112696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.112351    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"]
Nov 03 07:22:41.189192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.189169    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7141-e03248"
Nov 03 07:22:41.192339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.192295    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" podStartSLOduration=2.192281041 podStartE2EDuration="2.192281041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:41.190228293 +0000 UTC m=+1413.522597287" watchObservedRunningTime="2026-11-03 07:22:41.192281041 +0000 UTC m=+1413.524650035"
Nov 03 07:22:41.291833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.291803    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"]
Nov 03 07:22:41.296144 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.296114    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.315684 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.315659    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4b4f35-8784-4b3c-9f8f-e29409a82928-kube-api-access-fzlfd\" (UniqueName: \"kubernetes.io/projected/aa4b4f35-8784-4b3c-9f8f-e29409a82928-kube-api-access-fzlfd\") pod \"aa4b4f35-8784-4b3c-9f8f-e29409a82928\" (UID: \"aa4b4f35-8784-4b3c-9f8f-e29409a82928\") "
Nov 03 07:22:41.361173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.360498    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-33a46b37-gw-openshift-default-dockercfg-knw5s\""
Nov 03 07:22:41.385951 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.385917    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4b4f35-8784-4b3c-9f8f-e29409a82928-kube-api-access-fzlfd" pod "aa4b4f35-8784-4b3c-9f8f-e29409a82928" (UID: "aa4b4f35-8784-4b3c-9f8f-e29409a82928"). InnerVolumeSpecName "kube-api-access-fzlfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:41.404850 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.404820    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"]
Nov 03 07:22:41.417015 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.416988    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-credential-socket\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.417189 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.417059    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rddkx\" (UniqueName: \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-kube-api-access-rddkx\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.417275 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.417240    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-data\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.417334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.417274    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-envoy\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.417334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.417326    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06dd1225-38d7-4297-9c65-42e0788fbe3a-istiod-ca-cert\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.417487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.417388    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-certs\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.417487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.417412    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-socket\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.417487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.417444    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-token\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.417487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.417467    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-podinfo\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.417682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.417646    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzlfd\" (UniqueName: \"kubernetes.io/projected/aa4b4f35-8784-4b3c-9f8f-e29409a82928-kube-api-access-fzlfd\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:41.519379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.519300    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06dd1225-38d7-4297-9c65-42e0788fbe3a-istiod-ca-cert\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.519379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.519345    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-certs\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.519563 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.519365    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-socket\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.519563 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.519414    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-token\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.519563 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.519440    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-podinfo\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.519563 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.519534    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-credential-socket\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.519696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.519577    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rddkx\" (UniqueName: \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-kube-api-access-rddkx\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.519696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.519618    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-data\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.519696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.519648    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-envoy\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.520366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.520341    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-data\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.520485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.520372    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-credential-socket\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.520604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.520583    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-certs\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.520683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.520639    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-socket\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.520739 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.520707    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06dd1225-38d7-4297-9c65-42e0788fbe3a-istiod-ca-cert\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.522518 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.522494    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-envoy\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.522845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.522825    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-podinfo\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.584121 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.584089    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-token\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.584314 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.584287    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rddkx\" (UniqueName: \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-kube-api-access-rddkx\") pod \"e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") " pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.609719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.609685    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:41.737218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.737184    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:41.738599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.738574    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 07:22:41.738712 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.738620    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 07:22:41.820753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.820708    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4b4f35-8784-4b3c-9f8f-e29409a82928" path="/var/lib/kubelet/pods/aa4b4f35-8784-4b3c-9f8f-e29409a82928/volumes"
Nov 03 07:22:41.981091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.981059    4470 generic.go:356] "Generic (PLEG): container finished" podID="aa4b4f35-8784-4b3c-9f8f-e29409a82928" containerID="e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636" exitCode=137
Nov 03 07:22:41.981546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.981121    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7141-e03248"
Nov 03 07:22:41.981546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.981122    4470 scope.go:122] "RemoveContainer" containerID="e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636"
Nov 03 07:22:41.984017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.983993    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt" event={"ID":"e7e9755c-f3e3-4eae-ab90-fd646155683f","Type":"ContainerStarted","Data":"8d80225475e724b90a673b821569b59020f83a5cbb6e60d1adc4a4e728fde223"}
Nov 03 07:22:41.984160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.984043    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt" event={"ID":"e7e9755c-f3e3-4eae-ab90-fd646155683f","Type":"ContainerStarted","Data":"f6409c8d316a5b51e229fa63c7cd488a0c5e23fe2830683be1819a1e3101d19b"}
Nov 03 07:22:41.992797 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.992778    4470 scope.go:122] "RemoveContainer" containerID="e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636"
Nov 03 07:22:41.993094 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:22:41.993067    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636\": container with ID starting with e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636 not found: ID does not exist" containerID="e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636"
Nov 03 07:22:41.993158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:41.993109    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636"} err="failed to get container status \"e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636\": rpc error: code = NotFound desc = could not find container \"e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636\": container with ID starting with e1d35722585ae7db34c106c035ec407b08a95bb04fc4e39bdafd5f98caa64636 not found: ID does not exist"
Nov 03 07:22:42.106772 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:42.106742    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06dd1225_38d7_4297_9c65_42e0788fbe3a.slice/crio-56b16eec4557c336d93e9faf749c7754023eb19f4691a4ea25544d810c6a5369 WatchSource:0}: Error finding container 56b16eec4557c336d93e9faf749c7754023eb19f4691a4ea25544d810c6a5369: Status 404 returned error can't find the container with id 56b16eec4557c336d93e9faf749c7754023eb19f4691a4ea25544d810c6a5369
Nov 03 07:22:42.117302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.117260    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:42.117408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.117346    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:42.117408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.117388    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:42.117991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.117915    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"]
Nov 03 07:22:42.120522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.120479    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt" podStartSLOduration=2.120468029 podStartE2EDuration="2.120468029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:42.11308652 +0000 UTC m=+1414.445455513" watchObservedRunningTime="2026-11-03 07:22:42.120468029 +0000 UTC m=+1414.452837023"
Nov 03 07:22:42.178658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.178625    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:42.230746 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.230708    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea16936-8a96-44c1-a776-27c9c7cd6d70-kube-api-access-h6595\" (UniqueName: \"kubernetes.io/projected/8ea16936-8a96-44c1-a776-27c9c7cd6d70-kube-api-access-h6595\") pod \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") "
Nov 03 07:22:42.230877 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.230774    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-openshift-ca\") pod \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") "
Nov 03 07:22:42.230945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.230882    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-xks-ca\") pod \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\" (UID: \"8ea16936-8a96-44c1-a776-27c9c7cd6d70\") "
Nov 03 07:22:42.231432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.231381    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-openshift-ca" pod "8ea16936-8a96-44c1-a776-27c9c7cd6d70" (UID: "8ea16936-8a96-44c1-a776-27c9c7cd6d70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:42.231866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.231702    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:42.233407 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.233385    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea16936-8a96-44c1-a776-27c9c7cd6d70-kube-api-access-h6595" pod "8ea16936-8a96-44c1-a776-27c9c7cd6d70" (UID: "8ea16936-8a96-44c1-a776-27c9c7cd6d70"). InnerVolumeSpecName "kube-api-access-h6595". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:42.233886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.233850    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-xks-ca" pod "8ea16936-8a96-44c1-a776-27c9c7cd6d70" (UID: "8ea16936-8a96-44c1-a776-27c9c7cd6d70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:42.343931 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.342445    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6595\" (UniqueName: \"kubernetes.io/projected/8ea16936-8a96-44c1-a776-27c9c7cd6d70-kube-api-access-h6595\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:42.343931 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.342477    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea16936-8a96-44c1-a776-27c9c7cd6d70-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:42.736941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.736904    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 07:22:42.737122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.736947    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 07:22:42.994261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.994165    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" event={"ID":"06dd1225-38d7-4297-9c65-42e0788fbe3a","Type":"ContainerStarted","Data":"2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e"}
Nov 03 07:22:42.994261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.994235    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" event={"ID":"06dd1225-38d7-4297-9c65-42e0788fbe3a","Type":"ContainerStarted","Data":"56b16eec4557c336d93e9faf749c7754023eb19f4691a4ea25544d810c6a5369"}
Nov 03 07:22:42.995882 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.995856    4470 generic.go:356] "Generic (PLEG): container finished" podID="e7e9755c-f3e3-4eae-ab90-fd646155683f" containerID="8d80225475e724b90a673b821569b59020f83a5cbb6e60d1adc4a4e728fde223" exitCode=0
Nov 03 07:22:42.996008 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.995920    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt" event={"ID":"e7e9755c-f3e3-4eae-ab90-fd646155683f","Type":"ContainerDied","Data":"8d80225475e724b90a673b821569b59020f83a5cbb6e60d1adc4a4e728fde223"}
Nov 03 07:22:42.997379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.997362    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"
Nov 03 07:22:42.997467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.997407    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd" event={"ID":"8ea16936-8a96-44c1-a776-27c9c7cd6d70","Type":"ContainerDied","Data":"5de09ed29a4127198ebfeea3b51571b39029cebb665135a751b14745d6b481b6"}
Nov 03 07:22:42.997467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:42.997443    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5de09ed29a4127198ebfeea3b51571b39029cebb665135a751b14745d6b481b6"
Nov 03 07:22:43.145838 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.145786    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" podStartSLOduration=2.145772496 podStartE2EDuration="2.145772496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:43.142471489 +0000 UTC m=+1415.474840485" watchObservedRunningTime="2026-11-03 07:22:43.145772496 +0000 UTC m=+1415.478141490"
Nov 03 07:22:43.610624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.610592    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:43.616454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.616426    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:43.732651 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.732618    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"]
Nov 03 07:22:43.733327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.733311    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8ea16936-8a96-44c1-a776-27c9c7cd6d70" containerName="revoke-keys"
Nov 03 07:22:43.733385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.733328    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea16936-8a96-44c1-a776-27c9c7cd6d70" containerName="revoke-keys"
Nov 03 07:22:43.733468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.733458    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea16936-8a96-44c1-a776-27c9c7cd6d70" containerName="revoke-keys"
Nov 03 07:22:43.736874 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.736849    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 07:22:43.737058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.736899    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 07:22:43.737242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.737221    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:43.825432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.825404    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"]
Nov 03 07:22:43.896621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.896533    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7318aea9-kdsfz\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:43.896621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.896578    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7bts\" (UniqueName: \"kubernetes.io/projected/bf70f482-a8dc-4d16-84cb-07adad43e3e5-kube-api-access-m7bts\") pod \"maas-api-revoke-sub-7318aea9-kdsfz\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:43.896825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.896762    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7318aea9-kdsfz\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:43.998350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.998320    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7318aea9-kdsfz\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:43.998793 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.998496    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7318aea9-kdsfz\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:43.998793 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.998535    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7bts\" (UniqueName: \"kubernetes.io/projected/bf70f482-a8dc-4d16-84cb-07adad43e3e5-kube-api-access-m7bts\") pod \"maas-api-revoke-sub-7318aea9-kdsfz\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:43.999022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:43.999003    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7318aea9-kdsfz\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:44.001007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.000983    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7318aea9-kdsfz\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:44.005985 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.005963    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:44.007373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.007356    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:22:44.075816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.075755    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7bts\" (UniqueName: \"kubernetes.io/projected/bf70f482-a8dc-4d16-84cb-07adad43e3e5-kube-api-access-m7bts\") pod \"maas-api-revoke-sub-7318aea9-kdsfz\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:44.219175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.217852    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:44.375159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.374417    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:44.404881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.404849    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-xks-ca\") pod \"e7e9755c-f3e3-4eae-ab90-fd646155683f\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") "
Nov 03 07:22:44.405009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.404998    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e9755c-f3e3-4eae-ab90-fd646155683f-kube-api-access-vmjkq\" (UniqueName: \"kubernetes.io/projected/e7e9755c-f3e3-4eae-ab90-fd646155683f-kube-api-access-vmjkq\") pod \"e7e9755c-f3e3-4eae-ab90-fd646155683f\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") "
Nov 03 07:22:44.405571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.405547    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-openshift-ca\") pod \"e7e9755c-f3e3-4eae-ab90-fd646155683f\" (UID: \"e7e9755c-f3e3-4eae-ab90-fd646155683f\") "
Nov 03 07:22:44.405954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.405932    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-openshift-ca" pod "e7e9755c-f3e3-4eae-ab90-fd646155683f" (UID: "e7e9755c-f3e3-4eae-ab90-fd646155683f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:44.406177 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.406160    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:44.407420 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.407196    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-xks-ca" pod "e7e9755c-f3e3-4eae-ab90-fd646155683f" (UID: "e7e9755c-f3e3-4eae-ab90-fd646155683f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:44.407722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.407675    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e9755c-f3e3-4eae-ab90-fd646155683f-kube-api-access-vmjkq" pod "e7e9755c-f3e3-4eae-ab90-fd646155683f" (UID: "e7e9755c-f3e3-4eae-ab90-fd646155683f"). InnerVolumeSpecName "kube-api-access-vmjkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:44.507384 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.507354    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e9755c-f3e3-4eae-ab90-fd646155683f-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:44.507384 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.507378    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjkq\" (UniqueName: \"kubernetes.io/projected/e7e9755c-f3e3-4eae-ab90-fd646155683f-kube-api-access-vmjkq\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:44.737547 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.737474    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 07:22:44.737547 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.737522    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 07:22:44.780044 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:44.780011    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf70f482_a8dc_4d16_84cb_07adad43e3e5.slice/crio-2d97eb6d4877b5118f69c8d989fe8f6335b5d51d178d4d0c95fac792558ad3a6 WatchSource:0}: Error finding container 2d97eb6d4877b5118f69c8d989fe8f6335b5d51d178d4d0c95fac792558ad3a6: Status 404 returned error can't find the container with id 2d97eb6d4877b5118f69c8d989fe8f6335b5d51d178d4d0c95fac792558ad3a6
Nov 03 07:22:44.780410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:44.780390    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"]
Nov 03 07:22:45.015979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:45.015947    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz" event={"ID":"bf70f482-a8dc-4d16-84cb-07adad43e3e5","Type":"ContainerStarted","Data":"2d97eb6d4877b5118f69c8d989fe8f6335b5d51d178d4d0c95fac792558ad3a6"}
Nov 03 07:22:45.017400 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:45.017381    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"
Nov 03 07:22:45.017506 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:45.017424    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt" event={"ID":"e7e9755c-f3e3-4eae-ab90-fd646155683f","Type":"ContainerDied","Data":"f6409c8d316a5b51e229fa63c7cd488a0c5e23fe2830683be1819a1e3101d19b"}
Nov 03 07:22:45.017506 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:45.017469    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6409c8d316a5b51e229fa63c7cd488a0c5e23fe2830683be1819a1e3101d19b"
Nov 03 07:22:45.736452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:45.736417    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 07:22:45.736597 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:45.736464    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 07:22:46.027174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:46.027060    4470 generic.go:356] "Generic (PLEG): container finished" podID="bf70f482-a8dc-4d16-84cb-07adad43e3e5" containerID="7ec1b18769876bc96d15219a3ac3934549ac7d79abd3efadaf01366449b4b98d" exitCode=0
Nov 03 07:22:46.027174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:46.027150    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz" event={"ID":"bf70f482-a8dc-4d16-84cb-07adad43e3e5","Type":"ContainerDied","Data":"7ec1b18769876bc96d15219a3ac3934549ac7d79abd3efadaf01366449b4b98d"}
Nov 03 07:22:46.736650 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:46.736615    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 07:22:46.736838 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:46.736665    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 07:22:47.242065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.242034    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:47.269548 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.269503    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf70f482-a8dc-4d16-84cb-07adad43e3e5-kube-api-access-m7bts\" (UniqueName: \"kubernetes.io/projected/bf70f482-a8dc-4d16-84cb-07adad43e3e5-kube-api-access-m7bts\") pod \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") "
Nov 03 07:22:47.269733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.269703    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-openshift-ca\") pod \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") "
Nov 03 07:22:47.270206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.270177    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-openshift-ca" pod "bf70f482-a8dc-4d16-84cb-07adad43e3e5" (UID: "bf70f482-a8dc-4d16-84cb-07adad43e3e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:47.270328 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.270299    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-xks-ca\") pod \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\" (UID: \"bf70f482-a8dc-4d16-84cb-07adad43e3e5\") "
Nov 03 07:22:47.270942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.270915    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:47.271981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.271962    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf70f482-a8dc-4d16-84cb-07adad43e3e5-kube-api-access-m7bts" pod "bf70f482-a8dc-4d16-84cb-07adad43e3e5" (UID: "bf70f482-a8dc-4d16-84cb-07adad43e3e5"). InnerVolumeSpecName "kube-api-access-m7bts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:47.272842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.272819    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-xks-ca" pod "bf70f482-a8dc-4d16-84cb-07adad43e3e5" (UID: "bf70f482-a8dc-4d16-84cb-07adad43e3e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:47.376063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.376026    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf70f482-a8dc-4d16-84cb-07adad43e3e5-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:47.376063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.376065    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7bts\" (UniqueName: \"kubernetes.io/projected/bf70f482-a8dc-4d16-84cb-07adad43e3e5-kube-api-access-m7bts\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:47.741705 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.741624    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:47.742097 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.742076    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:47.742817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:47.742798    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:22:48.058209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:48.057547    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz" event={"ID":"bf70f482-a8dc-4d16-84cb-07adad43e3e5","Type":"ContainerDied","Data":"2d97eb6d4877b5118f69c8d989fe8f6335b5d51d178d4d0c95fac792558ad3a6"}
Nov 03 07:22:48.058209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:48.057609    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d97eb6d4877b5118f69c8d989fe8f6335b5d51d178d4d0c95fac792558ad3a6"
Nov 03 07:22:48.083157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:48.059479    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"
Nov 03 07:22:49.712166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.712122    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"]
Nov 03 07:22:49.712748 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.712735    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e7e9755c-f3e3-4eae-ab90-fd646155683f" containerName="revoke-keys"
Nov 03 07:22:49.712787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.712750    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e9755c-f3e3-4eae-ab90-fd646155683f" containerName="revoke-keys"
Nov 03 07:22:49.712836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.712788    4470 cpu_manager.go:414] "containerMap: removing container" podUID="bf70f482-a8dc-4d16-84cb-07adad43e3e5" containerName="revoke-keys"
Nov 03 07:22:49.712836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.712794    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf70f482-a8dc-4d16-84cb-07adad43e3e5" containerName="revoke-keys"
Nov 03 07:22:49.712902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.712893    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf70f482-a8dc-4d16-84cb-07adad43e3e5" containerName="revoke-keys"
Nov 03 07:22:49.712937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.712907    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e9755c-f3e3-4eae-ab90-fd646155683f" containerName="revoke-keys"
Nov 03 07:22:49.717169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.717127    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:49.773558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.773513    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:22:49.774823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.774802    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:22:49.776596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.776573    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"]
Nov 03 07:22:49.820714 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.820680    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:22:49.820885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.820763    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:22:49.859474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.859437    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99cf8fb-5rdhr\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:49.859623 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.859522    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99cf8fb-5rdhr\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:49.859682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.859667    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bh6r\" (UniqueName: \"kubernetes.io/projected/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-kube-api-access-5bh6r\") pod \"maas-api-revoke-sub-f99cf8fb-5rdhr\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:49.961340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.961304    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99cf8fb-5rdhr\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:49.961340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.961345    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99cf8fb-5rdhr\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:49.961602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.961467    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bh6r\" (UniqueName: \"kubernetes.io/projected/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-kube-api-access-5bh6r\") pod \"maas-api-revoke-sub-f99cf8fb-5rdhr\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:49.962063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.962043    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99cf8fb-5rdhr\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:49.963874 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:49.963817    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99cf8fb-5rdhr\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:50.026384 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:50.026352    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bh6r\" (UniqueName: \"kubernetes.io/projected/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-kube-api-access-5bh6r\") pod \"maas-api-revoke-sub-f99cf8fb-5rdhr\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:50.031566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:50.031541    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:50.419665 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:50.419636    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15758a2_9ff6_48d3_a8c7_8ecf60d41e3b.slice/crio-c716a274f1acb911d7bb8ea1770dbeba2fc1052e42986005b2b6ae2717b5ece6 WatchSource:0}: Error finding container c716a274f1acb911d7bb8ea1770dbeba2fc1052e42986005b2b6ae2717b5ece6: Status 404 returned error can't find the container with id c716a274f1acb911d7bb8ea1770dbeba2fc1052e42986005b2b6ae2717b5ece6
Nov 03 07:22:50.421458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:50.421433    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"]
Nov 03 07:22:51.087554 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:51.087517    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr" event={"ID":"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b","Type":"ContainerStarted","Data":"1aaa457ea24bafcd23260959694efdc03c1dd1b0f3f56ddb2b2409df05908d9a"}
Nov 03 07:22:51.087972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:51.087578    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr" event={"ID":"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b","Type":"ContainerStarted","Data":"c716a274f1acb911d7bb8ea1770dbeba2fc1052e42986005b2b6ae2717b5ece6"}
Nov 03 07:22:51.210841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:51.210780    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr" podStartSLOduration=2.210754809 podStartE2EDuration="2.210754809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:51.206622658 +0000 UTC m=+1423.538991661" watchObservedRunningTime="2026-11-03 07:22:51.210754809 +0000 UTC m=+1423.543123803"
Nov 03 07:22:52.097639 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:52.097605    4470 generic.go:356] "Generic (PLEG): container finished" podID="c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b" containerID="1aaa457ea24bafcd23260959694efdc03c1dd1b0f3f56ddb2b2409df05908d9a" exitCode=0
Nov 03 07:22:52.098039 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:52.097687    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr" event={"ID":"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b","Type":"ContainerDied","Data":"1aaa457ea24bafcd23260959694efdc03c1dd1b0f3f56ddb2b2409df05908d9a"}
Nov 03 07:22:53.266613 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.266589    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:53.306781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.306752    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-openshift-ca\") pod \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") "
Nov 03 07:22:53.306940 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.306849    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-xks-ca\") pod \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") "
Nov 03 07:22:53.307032 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.307013    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-kube-api-access-5bh6r\" (UniqueName: \"kubernetes.io/projected/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-kube-api-access-5bh6r\") pod \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\" (UID: \"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b\") "
Nov 03 07:22:53.307346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.307187    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-openshift-ca" pod "c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b" (UID: "c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:22:53.307745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.307574    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:53.309870 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.309845    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-xks-ca" pod "c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b" (UID: "c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:22:53.309943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.309889    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-kube-api-access-5bh6r" pod "c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b" (UID: "c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b"). InnerVolumeSpecName "kube-api-access-5bh6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:22:53.409046 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.408969    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bh6r\" (UniqueName: \"kubernetes.io/projected/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-kube-api-access-5bh6r\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:53.409046 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.408994    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:22:53.697402 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.697326    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"]
Nov 03 07:22:53.697969 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.697956    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b" containerName="revoke-keys"
Nov 03 07:22:53.698017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.697972    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b" containerName="revoke-keys"
Nov 03 07:22:53.698076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.698066    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b" containerName="revoke-keys"
Nov 03 07:22:53.701809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.701793    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.711984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.711958    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.712115 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.712076    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs6d5\" (UniqueName: \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-kube-api-access-qs6d5\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.712115 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.712103    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.712241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.712118    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.712335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.712320    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.712460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.712378    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.712460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.712445    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.712591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.712572    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ba92ef7-803c-4082-8641-c4d65e64b615-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.712654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.712622    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.736368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.736335    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"]
Nov 03 07:22:53.740652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.740634    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.789653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.789602    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"]
Nov 03 07:22:53.801582 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.801538    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"]
Nov 03 07:22:53.801773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.801650    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.808376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.808350    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.813967    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814027    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x7cx\" (UniqueName: \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-kube-api-access-6x7cx\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814058    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814090    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814122    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814187    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/368b3be0-0b24-4b53-a188-4920f1c46113-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814212    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814238    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814280    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814337    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs6d5\" (UniqueName: \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-kube-api-access-qs6d5\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814365    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814390    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814412    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814444    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l82g\" (UniqueName: \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-kube-api-access-8l82g\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814509    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814575    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814621    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814656    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814708    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814747    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814783    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814807    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814835    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814871    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814919    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814960    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/368b3be0-0b24-4b53-a188-4920f1c46113-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.814990    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ba92ef7-803c-4082-8641-c4d65e64b615-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.815822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.815816    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ba92ef7-803c-4082-8641-c4d65e64b615-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.819464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.816180    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.819464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.816486    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.819464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.816880    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.819464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.817387    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.832915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.832847    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.841935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.840867    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.846110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.846065    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"]
Nov 03 07:22:53.896492 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.896461    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"]
Nov 03 07:22:53.911193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.904279    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.911193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.906419    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs6d5\" (UniqueName: \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-kube-api-access-qs6d5\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:53.916612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.916181    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"]
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.917589    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.917694    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.917755    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.917834    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.917869    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.917900    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.917948    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.918012    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.918051    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/368b3be0-0b24-4b53-a188-4920f1c46113-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.918158    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7cx\" (UniqueName: \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-kube-api-access-6x7cx\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.918195    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.918236    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.918282    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/368b3be0-0b24-4b53-a188-4920f1c46113-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.918310    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.918336    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.918385    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.918461    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.919856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.918508    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l82g\" (UniqueName: \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-kube-api-access-8l82g\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.924774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.923656    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.924774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.923985    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.924774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.924261    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.924774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.924500    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.926398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.926370    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.927018 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.926991    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/368b3be0-0b24-4b53-a188-4920f1c46113-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.927018 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.927014    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.928246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.927684    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.928246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.927954    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.928246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.928091    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.928246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.928227    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.929186 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.928520    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.931105 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.931083    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"]
Nov 03 07:22:53.932736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.932698    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.938771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.938747    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/368b3be0-0b24-4b53-a188-4920f1c46113-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:53.994649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.994246    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:53.994649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:53.994590    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l82g\" (UniqueName: \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-kube-api-access-8l82g\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:54.002687 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.002652    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:54.004109 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.004082    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7cx\" (UniqueName: \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-kube-api-access-6x7cx\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:54.017930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.017892    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:54.019534 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.019505    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43aeb176-2074-480e-8ef5-251d3babd226-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.019670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.019555    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.019762 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.019744    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.019800 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.019783    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43aeb176-2074-480e-8ef5-251d3babd226-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.019845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.019803    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.019845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.019826    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.019972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.019954    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.020283 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.020256    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.020422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.020402    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjkkf\" (UniqueName: \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-kube-api-access-zjkkf\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.067832 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.067463    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:54.128879 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.128820    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43aeb176-2074-480e-8ef5-251d3babd226-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.129248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.129157    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.130170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.129448    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.130170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.129508    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43aeb176-2074-480e-8ef5-251d3babd226-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.130170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.129543    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.130170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.129582    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.130170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.129615    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.130170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.129742    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.130170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.129771    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkkf\" (UniqueName: \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-kube-api-access-zjkkf\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.132671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.131551    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43aeb176-2074-480e-8ef5-251d3babd226-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.132671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.131822    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.132671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.132347    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.132671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.132571    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.133481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.133089    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.133481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.133363    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:54.134699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.134676    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.136100 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.136061    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43aeb176-2074-480e-8ef5-251d3babd226-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.145494 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.145459    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr" event={"ID":"c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b","Type":"ContainerDied","Data":"c716a274f1acb911d7bb8ea1770dbeba2fc1052e42986005b2b6ae2717b5ece6"}
Nov 03 07:22:54.145579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.145528    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c716a274f1acb911d7bb8ea1770dbeba2fc1052e42986005b2b6ae2717b5ece6"
Nov 03 07:22:54.145660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.145605    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"
Nov 03 07:22:54.198632 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.198574    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkkf\" (UniqueName: \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-kube-api-access-zjkkf\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.200709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.200688    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.367848 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.361563    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"]
Nov 03 07:22:54.404685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.400167    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:22:54.413649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.413616    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"]
Nov 03 07:22:54.456055 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.456030    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-67db4bc0\""
Nov 03 07:22:54.456279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.456259    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-67db4bc0-dockercfg-ts2dd\""
Nov 03 07:22:54.464831 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:54.464802    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba92ef7_803c_4082_8641_c4d65e64b615.slice/crio-6f0c8b065d15dc4f823eef0cc7b84dfe289641e3bbfcec5c562808aa6e8e5d12 WatchSource:0}: Error finding container 6f0c8b065d15dc4f823eef0cc7b84dfe289641e3bbfcec5c562808aa6e8e5d12: Status 404 returned error can't find the container with id 6f0c8b065d15dc4f823eef0cc7b84dfe289641e3bbfcec5c562808aa6e8e5d12
Nov 03 07:22:54.467126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.467104    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"]
Nov 03 07:22:54.467676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.467645    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:54.467742 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.467707    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:54.467782 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.467737    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:54.492369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.492342    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw2zf\" (UniqueName: \"kubernetes.io/projected/3d978992-ceae-4b80-9c92-519ec5d9775b-kube-api-access-vw2zf\") pod \"payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9\" (UID: \"3d978992-ceae-4b80-9c92-519ec5d9775b\") " pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:22:54.492493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.492472    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d978992-ceae-4b80-9c92-519ec5d9775b-plugins-config-volume\") pod \"payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9\" (UID: \"3d978992-ceae-4b80-9c92-519ec5d9775b\") " pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:22:54.493428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.493405    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:54.573498 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:54.573462    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05b8fa9_f2cf_4809_8bea_47a84600c2b7.slice/crio-b3c307adce47793272fc341ddf1ab5f9a7685f6cb1cc8f4cc2f3be96d18bfd65 WatchSource:0}: Error finding container b3c307adce47793272fc341ddf1ab5f9a7685f6cb1cc8f4cc2f3be96d18bfd65: Status 404 returned error can't find the container with id b3c307adce47793272fc341ddf1ab5f9a7685f6cb1cc8f4cc2f3be96d18bfd65
Nov 03 07:22:54.576459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.576422    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:54.576550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.576516    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:54.576608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.576560    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:54.595022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.594585    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d978992-ceae-4b80-9c92-519ec5d9775b-plugins-config-volume\") pod \"payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9\" (UID: \"3d978992-ceae-4b80-9c92-519ec5d9775b\") " pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:22:54.595022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.594785    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw2zf\" (UniqueName: \"kubernetes.io/projected/3d978992-ceae-4b80-9c92-519ec5d9775b-kube-api-access-vw2zf\") pod \"payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9\" (UID: \"3d978992-ceae-4b80-9c92-519ec5d9775b\") " pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:22:54.595888 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.595865    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d978992-ceae-4b80-9c92-519ec5d9775b-plugins-config-volume\") pod \"payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9\" (UID: \"3d978992-ceae-4b80-9c92-519ec5d9775b\") " pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:22:54.598533 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.598488    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"]
Nov 03 07:22:54.604382 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:22:54.604353    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368b3be0_0b24_4b53_a188_4920f1c46113.slice/crio-71493da0631e2e834e1f481d77b744d1dcd8e63bb4aa4c9bc74e7b720b43fa50 WatchSource:0}: Error finding container 71493da0631e2e834e1f481d77b744d1dcd8e63bb4aa4c9bc74e7b720b43fa50: Status 404 returned error can't find the container with id 71493da0631e2e834e1f481d77b744d1dcd8e63bb4aa4c9bc74e7b720b43fa50
Nov 03 07:22:54.609058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.609019    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:54.609145 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.609108    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:54.609184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.609171    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:54.615175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.614757    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"]
Nov 03 07:22:54.662812 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.662765    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw2zf\" (UniqueName: \"kubernetes.io/projected/3d978992-ceae-4b80-9c92-519ec5d9775b-kube-api-access-vw2zf\") pod \"payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9\" (UID: \"3d978992-ceae-4b80-9c92-519ec5d9775b\") " pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:22:54.724501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.724451    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:22:54.889456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.889419    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:54.889568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.889456    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"]
Nov 03 07:22:54.889568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.889490    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:54.889568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:54.889523    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:22:55.162338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.162051    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"]
Nov 03 07:22:55.178608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.178280    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" event={"ID":"43aeb176-2074-480e-8ef5-251d3babd226","Type":"ContainerStarted","Data":"399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770"}
Nov 03 07:22:55.178608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.178341    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" event={"ID":"43aeb176-2074-480e-8ef5-251d3babd226","Type":"ContainerStarted","Data":"3ac66ad537a81d4d579f2a0d7521ddfe2c98362dab4c2ea8233e9da5db6bd263"}
Nov 03 07:22:55.180222 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.180178    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" event={"ID":"7ba92ef7-803c-4082-8641-c4d65e64b615","Type":"ContainerStarted","Data":"28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b"}
Nov 03 07:22:55.180353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.180229    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" event={"ID":"7ba92ef7-803c-4082-8641-c4d65e64b615","Type":"ContainerStarted","Data":"6f0c8b065d15dc4f823eef0cc7b84dfe289641e3bbfcec5c562808aa6e8e5d12"}
Nov 03 07:22:55.182066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.182040    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" event={"ID":"368b3be0-0b24-4b53-a188-4920f1c46113","Type":"ContainerStarted","Data":"f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d"}
Nov 03 07:22:55.182204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.182087    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" event={"ID":"368b3be0-0b24-4b53-a188-4920f1c46113","Type":"ContainerStarted","Data":"71493da0631e2e834e1f481d77b744d1dcd8e63bb4aa4c9bc74e7b720b43fa50"}
Nov 03 07:22:55.183922 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.183892    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" event={"ID":"a05b8fa9-f2cf-4809-8bea-47a84600c2b7","Type":"ContainerStarted","Data":"13e55407d1b8fc3cc16416ad64065a60fa1c9f115112dae1d3f161a9281e5d21"}
Nov 03 07:22:55.184007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.183930    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" event={"ID":"a05b8fa9-f2cf-4809-8bea-47a84600c2b7","Type":"ContainerStarted","Data":"b3c307adce47793272fc341ddf1ab5f9a7685f6cb1cc8f4cc2f3be96d18bfd65"}
Nov 03 07:22:55.312685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.312632    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podStartSLOduration=2.312618082 podStartE2EDuration="2.312618082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:55.308525067 +0000 UTC m=+1427.640894073" watchObservedRunningTime="2026-11-03 07:22:55.312618082 +0000 UTC m=+1427.644987076"
Nov 03 07:22:55.441263 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.441158    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podStartSLOduration=2.441124505 podStartE2EDuration="2.441124505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:55.43960167 +0000 UTC m=+1427.771970677" watchObservedRunningTime="2026-11-03 07:22:55.441124505 +0000 UTC m=+1427.773493498"
Nov 03 07:22:55.495453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.495379    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:22:55.497800 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.497770    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 07:22:55.498063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.497846    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 07:22:55.583165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.583020    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podStartSLOduration=2.582983711 podStartE2EDuration="2.582983711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:55.567665288 +0000 UTC m=+1427.900034283" watchObservedRunningTime="2026-11-03 07:22:55.582983711 +0000 UTC m=+1427.915352705"
Nov 03 07:22:55.722215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:55.719351    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podStartSLOduration=2.71933201 podStartE2EDuration="2.71933201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:55.693728845 +0000 UTC m=+1428.026097921" watchObservedRunningTime="2026-11-03 07:22:55.71933201 +0000 UTC m=+1428.051701005"
Nov 03 07:22:56.023542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.023444    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:22:56.024086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.024066    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 07:22:56.024280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.024258    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 07:22:56.074434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.072450    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:22:56.084514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.084482    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 07:22:56.084843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.084814    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 07:22:56.145098 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.145065    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:22:56.145500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.145475    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 07:22:56.145698 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.145674    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 07:22:56.228166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.226519    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9" event={"ID":"3d978992-ceae-4b80-9c92-519ec5d9775b","Type":"ContainerStarted","Data":"8126a5c604fddc0e4e00836af44fb21b22771018a408b5439dcc106918591d02"}
Nov 03 07:22:56.228166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.226577    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9" event={"ID":"3d978992-ceae-4b80-9c92-519ec5d9775b","Type":"ContainerStarted","Data":"8feb20ee3464991bbe76ebe023fefe1f08b408418abc8f0cde174d8f6f97aafa"}
Nov 03 07:22:56.256310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.240242    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:22:56.377795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.377734    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9" podStartSLOduration=2.377713926 podStartE2EDuration="2.377713926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:56.352736502 +0000 UTC m=+1428.685105497" watchObservedRunningTime="2026-11-03 07:22:56.377713926 +0000 UTC m=+1428.710082921"
Nov 03 07:22:56.505157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.503289    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 07:22:56.505157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.503351    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 07:22:56.753221 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.729456    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:56.949181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.944746    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 07:22:56.988160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:56.963369    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:57.034634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.034536    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 07:22:57.034897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.034868    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 07:22:57.047314 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.047266    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:57.133160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.118292    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 07:22:57.133160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.118371    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 07:22:57.154159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.140617    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 07:22:57.154159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.140690    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 07:22:57.332000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.331239    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:57.332000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.331676    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:22:57.398203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.383109    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:57.399534 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.387223    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:22:57.494380 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.494346    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 07:22:57.515845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.509210    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 07:22:57.714341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.714234    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:57.729782 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.729740    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:57.730577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.730237    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:22:57.956668 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.928767    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:57.956668 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.929631    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:57.956668 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.930666    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Nov 03 07:22:57.958641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:57.958602    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:58.143499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.143452    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:58.226157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.223206    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:22:58.226157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.225257    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 07:22:58.226157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.225324    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 07:22:58.226704 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.226679    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 07:22:58.226855 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.226835    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 07:22:58.227005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.226989    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 07:22:58.227107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.227090    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 07:22:58.228094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.228075    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:22:58.373161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.372355    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:58.373365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.373314    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:22:58.373365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.373347    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 03 07:22:58.436214 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.433493    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 07:22:58.461158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.459922    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:22:58.461158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.460058    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:22:58.500202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.475674    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:22:58.500202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.476046    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:22:58.746227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.733513    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:22:58.746227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.734179    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:22:58.840192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.781197    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:58.899071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.856634    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 03 07:22:58.948833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.948780    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:58.977110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:58.977065    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 03 07:22:59.019020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.009976    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 07:22:59.100371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.051497    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:22:59.100371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.092889    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:22:59.140165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.132560    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 07:22:59.172670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.158602    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 07:22:59.189231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.140470    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 07:22:59.189231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.183923    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 07:22:59.189231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.166381    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 07:22:59.189231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.183994    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 07:22:59.280299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.202537    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 07:22:59.280299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.202596    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 07:22:59.280299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.262207    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:59.280299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.264565    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 07:22:59.280299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.265859    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": context deadline exceeded"
Nov 03 07:22:59.376667 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.376346    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:59.406999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.406939    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:59.464158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.448486    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:59.464158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.448957    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 03 07:22:59.500325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.500280    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:22:59.592159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.579583    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:59.786948 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.786880    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:22:59.818173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.794252    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:23:00.020919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.952230    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:00.020919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:22:59.981225    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:00.050913 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.050821    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 07:23:00.051267 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.051198    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 07:23:00.079965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.079934    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 07:23:00.111346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.084085    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 07:23:00.210159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.157273    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 03 07:23:00.210159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.157310    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 07:23:00.210159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.157385    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 07:23:00.210159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.199302    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 07:23:00.262493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.262454    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:00.264752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.264705    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:00.302661 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.302494    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 07:23:00.303148 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.303080    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 07:23:00.376747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.376704    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:00.467719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.467678    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 03 07:23:00.468231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.468207    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 07:23:00.588215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.553628    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 07:23:00.588215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.579010    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:23:00.613303 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.613251    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 07:23:00.756160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.755599    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:23:00.827687 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.827627    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:00.894279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.875219    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:00.970166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:00.969952    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:01.075174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.060917    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 07:23:01.075174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.060980    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 07:23:01.075174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.071517    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 07:23:01.075174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.071567    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 07:23:01.145627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.145543    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 07:23:01.145871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.145840    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 07:23:01.177061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.177032    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"]
Nov 03 07:23:01.195282 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.194493    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:01.219726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.202427    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:01.219726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.216485    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:01.274058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.269526    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:01.274058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.269884    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": context deadline exceeded"
Nov 03 07:23:01.274058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.269968    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:01.331261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.331196    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"]
Nov 03 07:23:01.412954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.358553    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv5cc\" (UniqueName: \"kubernetes.io/projected/5d6538b2-8e39-461f-b0aa-95943a811c34-kube-api-access-rv5cc\") pod \"maas-api-revoke-sub-556526bc-jhbjr\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:01.446174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.426370    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-556526bc-jhbjr\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:01.446174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.426591    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-556526bc-jhbjr\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:01.446174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.388201    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:01.536211 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.488003    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:01.536211 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.488296    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 03 07:23:01.575421 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.575381    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 07:23:01.579281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.579232    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 07:23:01.600520 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.584217    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:01.600520 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.591112    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:01.622650 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.622611    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv5cc\" (UniqueName: \"kubernetes.io/projected/5d6538b2-8e39-461f-b0aa-95943a811c34-kube-api-access-rv5cc\") pod \"maas-api-revoke-sub-556526bc-jhbjr\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:01.622972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.622951    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-556526bc-jhbjr\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:01.623214 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.623195    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-556526bc-jhbjr\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:01.705542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.705444    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:23:01.705729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.705595    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:23:01.760981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.759065    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-556526bc-jhbjr\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:01.763215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.762357    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:23:01.763215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.762451    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:23:01.795782 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.795749    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-556526bc-jhbjr\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:01.856162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.846756    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv5cc\" (UniqueName: \"kubernetes.io/projected/5d6538b2-8e39-461f-b0aa-95943a811c34-kube-api-access-rv5cc\") pod \"maas-api-revoke-sub-556526bc-jhbjr\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:01.924161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.910701    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:01.924161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.911502    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:01.975217 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:01.972788    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:02.054169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.027754    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 07:23:02.054169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.027811    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 07:23:02.084171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.084122    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 07:23:02.084461 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.084426    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 07:23:02.156192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.140478    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:02.208159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.157758    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 07:23:02.208159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.192249    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 07:23:02.208159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.194930    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:02.221167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.217240    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:02.296095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.277224    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 07:23:02.296095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.277591    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 07:23:02.428237 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.428195    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:02.493867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.493832    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 07:23:02.494095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.494071    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 07:23:02.494297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.494278    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:02.494623 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.494603    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 07:23:02.598811 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.598773    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:02.845420 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.843151    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:23:02.873218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.871481    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:23:02.906918 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.900617    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:23:02.912432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:02.912381    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:23:03.143888 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.143800    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 07:23:03.145388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.145357    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 07:23:03.145570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.144688    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 07:23:03.145692 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.145674    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 07:23:03.147295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.146831    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:03.201955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.179862    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 07:23:03.204466 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.204439    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 07:23:03.204671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.204342    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:03.217573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.217530    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:03.596216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.595530    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:03.661161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.630533    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 07:23:03.693935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.693899    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 07:23:03.698288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.698257    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 07:23:03.736289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.736238    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:23:03.749381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.749330    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:23:03.812158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.791786    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:03.850216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.822076    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Nov 03 07:23:03.998653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:03.979625    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:04.208553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.208459    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 07:23:04.208815 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.208786    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 07:23:04.268160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.260453    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:04.293922 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.293883    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 07:23:04.294225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.294193    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 07:23:04.329155 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.300782    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Nov 03 07:23:04.329155 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.326778    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:04.333369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.333333    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 07:23:04.333630 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.333589    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 07:23:04.583164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.565256    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 07:23:04.865435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.865386    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 07:23:04.865697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.672186    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:04.865898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.669563    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:04.866092 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.782289    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 07:23:04.888158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.782392    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 07:23:04.889059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.830821    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:04.917157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.830905    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:23:04.996203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:04.782487    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:05.040358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.040284    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:05.114722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.114623    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 07:23:05.114968 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.114937    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 07:23:05.127234 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:05.126859    4470 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.303s"
Nov 03 07:23:05.277929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.277586    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:05.318768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.318732    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 07:23:05.319031 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.319002    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 07:23:05.334192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.322813    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Nov 03 07:23:05.335430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.335394    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Nov 03 07:23:05.371910 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.371825    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:23:05.560160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.540446    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 07:23:05.572836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.572799    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 07:23:05.728077 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:23:05.728043    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-8f491347a69930cad3077f7ece3cf80b3a090f40a62742a45aac9b55fdfd3fec WatchSource:0}: Error finding container 8f491347a69930cad3077f7ece3cf80b3a090f40a62742a45aac9b55fdfd3fec: Status 404 returned error can't find the container with id 8f491347a69930cad3077f7ece3cf80b3a090f40a62742a45aac9b55fdfd3fec
Nov 03 07:23:05.898410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.898362    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:05.899584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.899551    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:05.900075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.900055    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:23:05.908699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.908662    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 03 07:23:05.934161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.912041    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 03 07:23:06.308977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:05.937000    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:06.309887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.038635    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:06.310319 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.038670    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:23:06.310663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.041222    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:06.310988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.161289    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:06.311158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.311110    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:06.311473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.278269    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:06.319090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.318961    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:06.319090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.319017    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:06.346103 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.346053    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:06.347227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.347196    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:06.644377 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:06.644335    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:23:06.747157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.731530    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": context deadline exceeded"
Nov 03 07:23:07.111544 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.900483    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:23:07.111794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.111769    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr" event={"ID":"5d6538b2-8e39-461f-b0aa-95943a811c34","Type":"ContainerStarted","Data":"8f491347a69930cad3077f7ece3cf80b3a090f40a62742a45aac9b55fdfd3fec"}
Nov 03 07:23:07.111942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.922291    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 07:23:07.118241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.922328    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 03 07:23:07.119597 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:06.922251    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 07:23:07.148010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.147973    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:07.148321 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.148287    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:07.251364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.251312    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:07.251704 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.251652    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:07.279700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.279591    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:07.279700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.279660    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:07.294184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.288994    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"]
Nov 03 07:23:07.317618 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.317562    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:07.319588 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.319536    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 07:23:07.320866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.320823    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:23:07.384784 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.337939    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:23:07.384784 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.340465    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:23:07.384784 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.361814    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Nov 03 07:23:07.384784 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.363543    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Nov 03 07:23:07.386722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.386680    4470 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:07.386985 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.386934    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:07.810312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.810268    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:07.951918 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.951828    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:07.952194 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.952165    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:07.991168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.979401    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:07.991486 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:07.991448    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.036588 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.036555    4470 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:08.036902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.036853    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.098304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.094370    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:08.098304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.094444    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.098304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.098095    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": context deadline exceeded"
Nov 03 07:23:08.100518 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.100311    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:08.100518 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.100372    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.129665 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.129607    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 07:23:08.263167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.259908    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:08.263167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.259965    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.263167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.263065    4470 scope.go:122] "RemoveContainer" containerID="7d03e13dd368218146b226e7d0fbdd841f8d9d7ba87bab06fe40af902d8105bb"
Nov 03 07:23:08.320732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.320597    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.323387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.323350    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.332466 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.332427    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.336191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.332911    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 03 07:23:08.365703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.365661    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.392160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.367248    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:23:08.392792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.374548    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Nov 03 07:23:08.427161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.406427    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:08.493237 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.493195    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.493987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.447228    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:08.494199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.494174    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.607108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.500678    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:08.798762 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.729835    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:08.899880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.716185    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:23:08.901155 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.716047    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:23:08.902212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.808660    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 03 07:23:08.941597 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:08.941562    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:23:09.183221 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.133895    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:09.414226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.301521    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 03 07:23:09.414226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.342755    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 07:23:09.414226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.343271    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:09.414226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.361813    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:23:09.414226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.367713    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:09.414226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.394995    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:09.486647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.486542    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:09.688164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.509361    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:09.688625 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.514565    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"]
Nov 03 07:23:09.688788 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.606750    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:23:09.939159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:09.894945    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:10.104033 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.046024    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Nov 03 07:23:10.104033 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.046578    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded"
Nov 03 07:23:10.104033 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.047017    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:23:10.104033 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.057076    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 03 07:23:10.129048 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.129013    4470 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:10.129341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.129312    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:10.248908 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.248807    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:10.386539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.374025    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:10.419155 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.417940    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 07:23:10.419155 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.418267    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 07:23:10.442754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.441773    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Nov 03 07:23:10.472341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.472277    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Nov 03 07:23:10.567675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.567590    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2" podUID="dd115615-a8ce-4efa-bc4f-5d32c851601f" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 07:23:10.627645 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.612822    4470 scope.go:122] "RemoveContainer" containerID="b62a6030d64f786bfb6a7a876255f69dd0c47a2b4159ce1a71424c5eb3df92f9"
Nov 03 07:23:10.627645 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.613020    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:10.652159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.629184    4470 patch_prober.go:29] interesting pod/thanos-querier-7d4997f974-cg8jj container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:10.694365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.694319    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-7d4997f974-cg8jj" podUID="440f457a-b678-4b70-b5cd-07d53dbbef71" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:10.694670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.640737    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:10.916213 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:10.910771    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:11.062339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.062301    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:11.103763 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.103730    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.167215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.151887    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:23:11.167215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.162019    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 07:23:11.187167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.185508    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 07:23:11.187167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.185601    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 07:23:11.187167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.185666    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 07:23:11.187167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.185792    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zrhnm\""
Nov 03 07:23:11.262266 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.262213    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": context deadline exceeded"
Nov 03 07:23:11.298281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.265895    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:23:11.338758 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.338717    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.339126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.339109    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.339555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.339528    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.339781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.339765    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.339985 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.339968    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjfxl\" (UniqueName: \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-kube-api-access-kjfxl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.340273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.340256    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.804442 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.370657    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:23:11.805120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.402559    4470 scope.go:122] "RemoveContainer" containerID="d5f3c232578c6811a2074a8759d9f7be2fe7171b76b3a8a0ad26556b6a013476"
Nov 03 07:23:11.809173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.404124    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:11.809388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.424230    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 07:23:11.809531 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.513229    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:11.809673 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.536737    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr" podStartSLOduration=12.536715453 podStartE2EDuration="12.536715453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:11.263362234 +0000 UTC m=+1443.595731228" watchObservedRunningTime="2026-11-03 07:23:11.536715453 +0000 UTC m=+1443.869084448"
Nov 03 07:23:11.819820 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.547273    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:11.820048 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.160042    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:11.820227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.615358    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:11.820374 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.768404    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:11.820555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.820535    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.820964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.820946    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:23:11.835483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.835463    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:23:11.835685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.835672    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"]
Nov 03 07:23:11.840205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.840177    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:23:11.840562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.840235    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:23:11.840562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.840260    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:23:11.840562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.840286    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:23:11.941601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.941563    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfxl\" (UniqueName: \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-kube-api-access-kjfxl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.941896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.941872    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.942056 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.942040    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.942309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.942270    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.942536 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.942516    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.943188 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.943166    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:11.943428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:11.943407    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:12.216822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.213689    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:12.216822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.214898    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfxl\" (UniqueName: \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-kube-api-access-kjfxl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:12.216822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.215218    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:12.231860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.224816    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:12.231860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.227153    4470 generic.go:356] "Generic (PLEG): container finished" podID="5d6538b2-8e39-461f-b0aa-95943a811c34" containerID="6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca" exitCode=0
Nov 03 07:23:12.231860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.227916    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:12.232778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.232755    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:12.238958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.238926    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr" event={"ID":"5d6538b2-8e39-461f-b0aa-95943a811c34","Type":"ContainerDied","Data":"6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca"}
Nov 03 07:23:12.239080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239011    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:23:12.239172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239082    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:23:12.239172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239120    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:23:12.239286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239176    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:23:12.239286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239228    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:23:12.239286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239262    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:23:12.239441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239296    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:23:12.239441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239326    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:23:12.239441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239356    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:23:12.239441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239389    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:23:12.239441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239417    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:23:12.239720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239450    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:23:12.239720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239486    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:23:12.239720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.239517    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:23:12.240514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.240491    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47mcc\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:12.262562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.262395    4470 scope.go:122] "RemoveContainer" containerID="ab8283ed7096301da506b0a9cb24cc2321de5e44f2241ffcf2b0de32bf2a8cc3"
Nov 03 07:23:12.272536 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.272514    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:12.300108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.300073    4470 scope.go:122] "RemoveContainer" containerID="340f8d2523b839898019d84ec470fa0cca2e01796e6b3135548d04b2ac436f86"
Nov 03 07:23:12.303675 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:12.303589    4470 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-conmon-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 07:23:12.373549 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.373521    4470 scope.go:122] "RemoveContainer" containerID="5d5b1cedeb22f29b8f7d31b5729b66ea919d7c4647dd0b2069d904f7b79d8d43"
Nov 03 07:23:12.389368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.389343    4470 scope.go:122] "RemoveContainer" containerID="50ccb3ad3ccfc2d76a5355f973d193646c197b3c2f37bb4469272bb9277c4547"
Nov 03 07:23:12.408897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.408870    4470 scope.go:122] "RemoveContainer" containerID="2c991703d9a25f0f5a0488bd3ade7e52d95cb227b3c2814ecfb70eaa6324d234"
Nov 03 07:23:12.442867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.442833    4470 scope.go:122] "RemoveContainer" containerID="bb8a8cd5177db1f978c5107ffab525febb76dde1b8456020773a6655831d9949"
Nov 03 07:23:12.461372 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.461292    4470 scope.go:122] "RemoveContainer" containerID="af6b1667b978f22265cf5131c7df9ee370335e09f521faff2a3116c7ec9bd69d"
Nov 03 07:23:12.483527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.482895    4470 scope.go:122] "RemoveContainer" containerID="1516d0835e9c35de25e110df2802cbb29452448d237ab191c739ffdecc9a81f2"
Nov 03 07:23:12.496613 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.496596    4470 scope.go:122] "RemoveContainer" containerID="3ce29e94d1273d082581a33968d8b09f984bef08ecf76d1ea72f3654def68ce1"
Nov 03 07:23:12.784418 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:23:12.784387    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb31eb4_c728_4128_a2bd_a3ed99aaf99c.slice/crio-300f2c53d8b0fb10063f6b78489b8b9ee3931f07a5489447d2362c54b3e38ecc WatchSource:0}: Error finding container 300f2c53d8b0fb10063f6b78489b8b9ee3931f07a5489447d2362c54b3e38ecc: Status 404 returned error can't find the container with id 300f2c53d8b0fb10063f6b78489b8b9ee3931f07a5489447d2362c54b3e38ecc
Nov 03 07:23:12.786625 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:12.786599    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"]
Nov 03 07:23:13.250272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.250236    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc" event={"ID":"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c","Type":"ContainerStarted","Data":"300f2c53d8b0fb10063f6b78489b8b9ee3931f07a5489447d2362c54b3e38ecc"}
Nov 03 07:23:13.254074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.254048    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:23:13.442824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.442772    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:13.553774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.553684    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-openshift-ca\") pod \"5d6538b2-8e39-461f-b0aa-95943a811c34\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") "
Nov 03 07:23:13.553774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.553767    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6538b2-8e39-461f-b0aa-95943a811c34-kube-api-access-rv5cc\" (UniqueName: \"kubernetes.io/projected/5d6538b2-8e39-461f-b0aa-95943a811c34-kube-api-access-rv5cc\") pod \"5d6538b2-8e39-461f-b0aa-95943a811c34\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") "
Nov 03 07:23:13.554084 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.554061    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-openshift-ca" pod "5d6538b2-8e39-461f-b0aa-95943a811c34" (UID: "5d6538b2-8e39-461f-b0aa-95943a811c34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:23:13.554167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.554066    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-xks-ca\") pod \"5d6538b2-8e39-461f-b0aa-95943a811c34\" (UID: \"5d6538b2-8e39-461f-b0aa-95943a811c34\") "
Nov 03 07:23:13.554822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.554793    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:23:13.557471 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.557443    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6538b2-8e39-461f-b0aa-95943a811c34-kube-api-access-rv5cc" pod "5d6538b2-8e39-461f-b0aa-95943a811c34" (UID: "5d6538b2-8e39-461f-b0aa-95943a811c34"). InnerVolumeSpecName "kube-api-access-rv5cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:23:13.559183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.559150    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-xks-ca" pod "5d6538b2-8e39-461f-b0aa-95943a811c34" (UID: "5d6538b2-8e39-461f-b0aa-95943a811c34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:23:13.656599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.656549    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d6538b2-8e39-461f-b0aa-95943a811c34-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:23:13.656599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:13.656584    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv5cc\" (UniqueName: \"kubernetes.io/projected/5d6538b2-8e39-461f-b0aa-95943a811c34-kube-api-access-rv5cc\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:23:14.261949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:14.261875    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"
Nov 03 07:23:14.261949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:14.261887    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr" event={"ID":"5d6538b2-8e39-461f-b0aa-95943a811c34","Type":"ContainerDied","Data":"8f491347a69930cad3077f7ece3cf80b3a090f40a62742a45aac9b55fdfd3fec"}
Nov 03 07:23:14.261949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:14.261942    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f491347a69930cad3077f7ece3cf80b3a090f40a62742a45aac9b55fdfd3fec"
Nov 03 07:23:15.089046 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:15.089008    4470 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-conmon-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 07:23:15.089254 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:15.089020    4470 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-conmon-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 07:23:16.719305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:16.719272    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:23:16.719601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:16.719333    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:23:17.292298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:17.292267    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc" event={"ID":"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c","Type":"ContainerStarted","Data":"a0b905d331b6f0ac173c904c8a6d34dbcc52281b09e30a5c42d4437c28883031"}
Nov 03 07:23:17.292603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:17.292581    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:17.294193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:17.294170    4470 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-47mcc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:23:17.294310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:17.294214    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc" podUID="9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:23:17.423527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:17.423459    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc" podStartSLOduration=6.491915893 podStartE2EDuration="10.423440051s" totalImagesPullingTime="3.931524158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:17.418229248 +0000 UTC m=+1449.750598236" watchObservedRunningTime="2026-11-03 07:23:17.423440051 +0000 UTC m=+1449.755809046"
Nov 03 07:23:18.303588 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:18.303555    4470 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-47mcc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:23:18.303962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:18.303602    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc" podUID="9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:23:21.051480 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:21.051444    4470 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-conmon-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 07:23:21.303954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:21.303877    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:23:27.339691 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:27.339658    4470 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-conmon-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 07:23:31.065987 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:31.065949    4470 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-conmon-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 07:23:33.536400 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.536369    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"]
Nov 03 07:23:33.537049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.537035    4470 cpu_manager.go:414] "containerMap: removing container" podUID="5d6538b2-8e39-461f-b0aa-95943a811c34" containerName="revoke-keys"
Nov 03 07:23:33.537093 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.537051    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6538b2-8e39-461f-b0aa-95943a811c34" containerName="revoke-keys"
Nov 03 07:23:33.537211 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.537198    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6538b2-8e39-461f-b0aa-95943a811c34" containerName="revoke-keys"
Nov 03 07:23:33.542377 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.542360    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.611847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.611803    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89ggc\""
Nov 03 07:23:33.656382 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.656343    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"]
Nov 03 07:23:33.684467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.684427    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv4vf\" (UniqueName: \"kubernetes.io/projected/cd812ea7-1e46-432d-acf9-ff1081a5863f-kube-api-access-jv4vf\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.684676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.684531    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.684757 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.684661    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.684817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.684783    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.684902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.684881    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.685045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.685024    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.685115 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.685083    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.685236 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.685206    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.685302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.685280    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd812ea7-1e46-432d-acf9-ff1081a5863f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.786276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.786236    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd812ea7-1e46-432d-acf9-ff1081a5863f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.786503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.786347    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv4vf\" (UniqueName: \"kubernetes.io/projected/cd812ea7-1e46-432d-acf9-ff1081a5863f-kube-api-access-jv4vf\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.786503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.786395    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.786503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.786471    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.786691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.786517    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.786691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.786580    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.786799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.786692    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.786943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.786897    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.787057 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.786936    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.787057 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.786958    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.787057 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.787025    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd812ea7-1e46-432d-acf9-ff1081a5863f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.787394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.787373    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.787661 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.787638    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.787709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.787669    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.789341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.789318    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.789651 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.789631    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.853546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.853509    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd812ea7-1e46-432d-acf9-ff1081a5863f-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.853696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.853604    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv4vf\" (UniqueName: \"kubernetes.io/projected/cd812ea7-1e46-432d-acf9-ff1081a5863f-kube-api-access-jv4vf\") pod \"maas-default-gateway-openshift-default-58d898bd55-kx6fw\" (UID: \"cd812ea7-1e46-432d-acf9-ff1081a5863f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:33.857452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:33.857435    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:34.298850 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:23:34.298822    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd812ea7_1e46_432d_acf9_ff1081a5863f.slice/crio-5aae3f70941dae1858b615c5c7b452ad55a3b6e8f4eb9c985839522be671d284 WatchSource:0}: Error finding container 5aae3f70941dae1858b615c5c7b452ad55a3b6e8f4eb9c985839522be671d284: Status 404 returned error can't find the container with id 5aae3f70941dae1858b615c5c7b452ad55a3b6e8f4eb9c985839522be671d284
Nov 03 07:23:34.301295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:34.301272    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"]
Nov 03 07:23:34.302106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:34.302068    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:23:34.302217 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:34.302157    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:23:34.302217 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:34.302187    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:23:34.527731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:34.527694    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" event={"ID":"cd812ea7-1e46-432d-acf9-ff1081a5863f","Type":"ContainerStarted","Data":"bba1fd308ce0e604e28d5aa7785921b9185f7125ef17bdc29c42e5644a8e4f42"}
Nov 03 07:23:34.527934 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:34.527762    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" event={"ID":"cd812ea7-1e46-432d-acf9-ff1081a5863f","Type":"ContainerStarted","Data":"5aae3f70941dae1858b615c5c7b452ad55a3b6e8f4eb9c985839522be671d284"}
Nov 03 07:23:34.656873 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:34.656822    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" podStartSLOduration=1.656805725 podStartE2EDuration="1.656805725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:34.653942622 +0000 UTC m=+1466.986311618" watchObservedRunningTime="2026-11-03 07:23:34.656805725 +0000 UTC m=+1466.989174718"
Nov 03 07:23:34.766221 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:34.766175    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:23:34.857720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:34.857690    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:35.875159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:35.869433    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 07:23:35.875159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:35.869494    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" podUID="cd812ea7-1e46-432d-acf9-ff1081a5863f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 07:23:36.862598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:36.862558    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 07:23:36.862941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:36.862911    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" podUID="cd812ea7-1e46-432d-acf9-ff1081a5863f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 07:23:37.865160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:37.864516    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 07:23:37.865160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:37.864576    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" podUID="cd812ea7-1e46-432d-acf9-ff1081a5863f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 07:23:38.871397 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:38.871354    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 07:23:38.871928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:38.871436    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" podUID="cd812ea7-1e46-432d-acf9-ff1081a5863f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 07:23:39.244701 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.244605    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"]
Nov 03 07:23:39.259703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.259671    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.307546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.307514    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6sb9\" (UniqueName: \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-kube-api-access-p6sb9\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.307705 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.307569    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.307705 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.307614    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.307795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.307734    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.307795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.307761    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.307795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.307785    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.307963 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.307951    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.308007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.307996    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.308102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.308084    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.398797 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.398763    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"]
Nov 03 07:23:39.411119 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.411084    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.411461 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.411241    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.411461 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.411334    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.411678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.411443    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6sb9\" (UniqueName: \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-kube-api-access-p6sb9\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.411678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.411522    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.411678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.411595    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.411678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.411648    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.411895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.411681    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.411895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.411730    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.411895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.411761    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.412347 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.412229    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.416496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.413691    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.416496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.413947    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.416496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.414432    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.420893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.420865    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.426468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.426443    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.479264 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.479200    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6sb9\" (UniqueName: \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-kube-api-access-p6sb9\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.482961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.482866    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.588884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.588847    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:39.874458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.874006    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 07:23:39.874458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:39.874064    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" podUID="cd812ea7-1e46-432d-acf9-ff1081a5863f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 07:23:40.077722 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:23:40.077691    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92a8b3b_a5fb_48c6_945e_8e09b712cbdb.slice/crio-2d61055a53a0eff278789e31f38b2495ff502810e43cb24b19b6cd49aed99502 WatchSource:0}: Error finding container 2d61055a53a0eff278789e31f38b2495ff502810e43cb24b19b6cd49aed99502: Status 404 returned error can't find the container with id 2d61055a53a0eff278789e31f38b2495ff502810e43cb24b19b6cd49aed99502
Nov 03 07:23:40.080387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:40.080264    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"]
Nov 03 07:23:40.083723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:40.082312    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:23:40.083723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:40.082403    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:23:40.083723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:40.082453    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:23:40.604903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:40.604871    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" event={"ID":"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb","Type":"ContainerStarted","Data":"d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b"}
Nov 03 07:23:40.605084 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:40.604963    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" event={"ID":"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb","Type":"ContainerStarted","Data":"2d61055a53a0eff278789e31f38b2495ff502810e43cb24b19b6cd49aed99502"}
Nov 03 07:23:40.783159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:40.783018    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" podStartSLOduration=1.782996501 podStartE2EDuration="1.782996501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:40.734517744 +0000 UTC m=+1473.066886738" watchObservedRunningTime="2026-11-03 07:23:40.782996501 +0000 UTC m=+1473.115365496"
Nov 03 07:23:40.886239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:40.878318    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 07:23:40.886239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:40.878386    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" podUID="cd812ea7-1e46-432d-acf9-ff1081a5863f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 07:23:41.292994 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:41.292909    4470 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-conmon-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 07:23:41.589479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:41.589446    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:41.591403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:41.591382    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 07:23:41.591502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:41.591428    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 07:23:41.861610 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:41.861521    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 07:23:41.861873 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:41.861844    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" podUID="cd812ea7-1e46-432d-acf9-ff1081a5863f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 07:23:42.271419 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:42.271324    4470 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-conmon-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 07:23:42.589753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:42.589718    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 07:23:42.589995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:42.589967    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 07:23:43.382156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:43.381918    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:43.383356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:43.383332    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:43.384185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:43.384163    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw"
Nov 03 07:23:43.589803 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:43.589773    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 07:23:43.589956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:43.589823    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 07:23:44.617239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:44.609840    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 07:23:44.617239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:44.609898    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 07:23:45.589219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:45.589188    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 07:23:45.589370 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:45.589249    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 07:23:46.590109 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:46.590076    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 07:23:46.590655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:46.590159    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 07:23:47.645962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:47.645928    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:47.647220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:47.647117    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:47.648443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:47.648413    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:23:51.377773 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:51.377688    4470 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-conmon-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 07:23:58.057463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:57.982308    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:58.134276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.134223    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:58.191557 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.191509    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:58.192246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.192215    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:58.287694 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.287648    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:58.353288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.353239    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:58.397219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.397176    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:58.416486 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.415267    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:58.436593 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:23:58.436550    4470 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-conmon-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 07:23:58.476778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.476222    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:58.564930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.564325    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:23:58.806410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.806290    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 03 07:23:58.817290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.817217    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:59.009369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:58.996247    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:59.156158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.155342    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:59.156158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.155950    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:23:59.196158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.180591    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:59.200020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.198621    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:59.235220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.207601    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:23:59.258628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.258587    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:59.274106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.263246    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:23:59.310794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.310738    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 07:23:59.313344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.312417    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:23:59.432884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.378755    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 07:23:59.432884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.379117    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:23:59.432884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.400221    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:59.432884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.404396    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:23:59.432884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.418265    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:59.438158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.436556    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 03 07:23:59.438158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.437011    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:23:59.554508 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.554306    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:23:59.554841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.554810    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" podUID="cd812ea7-1e46-432d-acf9-ff1081a5863f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:59.559065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.559026    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Nov 03 07:23:59.561476 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.561456    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:23:59.565833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.565808    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:59.567069 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.567046    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded"
Nov 03 07:23:59.567301 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.567282    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:23:59.727380 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.727281    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:23:59.843154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:23:59.838470    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:24:00.009468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:00.009387    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:24:00.009724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:00.009706    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:24:00.009921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:00.009907    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:24:00.010344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:00.010329    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:24:00.010513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:00.010501    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:24:00.011382 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:00.010732    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:24:00.011539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:00.011526    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:24:00.012416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:00.012401    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:24:00.012579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:00.012566    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:24:00.012711 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:00.012699    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:24:00.012836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:00.012824    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:24:01.398409 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:24:01.398372    4470 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-conmon-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6538b2_8e39_461f_b0aa_95943a811c34.slice/crio-6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 07:24:04.761038 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:24:04.761001    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:24:12.664919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:12.664886    4470 scope.go:122] "RemoveContainer" containerID="70dd99aa8848dde63457ac364cea9609666c6d14170d5904d532bb6594feb122"
Nov 03 07:24:12.675502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:12.675482    4470 scope.go:122] "RemoveContainer" containerID="972092f0dfc2cdf094b45a999ba4ed88d31ddee9da7cfdcdc165d041a03f5fb9"
Nov 03 07:24:12.689103 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:12.689073    4470 scope.go:122] "RemoveContainer" containerID="cdf924c841627b06eb5539755aa98b4637186586873046e7f8a6ec16ee8afd24"
Nov 03 07:24:18.881814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:18.881775    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"]
Nov 03 07:24:18.888343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:18.888318    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.045184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.045098    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b225a1-0e3c-48c9-9997-46b62edb5072-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.045184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.045166    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229kb\" (UniqueName: \"kubernetes.io/projected/f9b225a1-0e3c-48c9-9997-46b62edb5072-kube-api-access-229kb\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.045455 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.045279    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.045455 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.045330    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.045611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.045583    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.045718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.045699    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.045782 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.045746    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.045839 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.045816    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.045902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.045840    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.111561 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.111528    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"]
Nov 03 07:24:19.147684 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.147577    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.147858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.147696    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.147858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.147740    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.147858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.147832    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.148051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.147868    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.148051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.148026    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b225a1-0e3c-48c9-9997-46b62edb5072-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.148051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.148032    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.148246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.148074    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229kb\" (UniqueName: \"kubernetes.io/projected/f9b225a1-0e3c-48c9-9997-46b62edb5072-kube-api-access-229kb\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.148246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.148089    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.148246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.148125    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.148246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.148197    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.148459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.148325    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.148657 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.148628    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.148794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.148770    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b225a1-0e3c-48c9-9997-46b62edb5072-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.150850 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.150821    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.151259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.151240    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.214492 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.214463    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b225a1-0e3c-48c9-9997-46b62edb5072-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.214777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.214751    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-229kb\" (UniqueName: \"kubernetes.io/projected/f9b225a1-0e3c-48c9-9997-46b62edb5072-kube-api-access-229kb\") pod \"maas-default-gateway-openshift-default-58d898bd55-pmhtb\" (UID: \"f9b225a1-0e3c-48c9-9997-46b62edb5072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.504816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.504731    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:19.944551 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:24:19.944521    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b225a1_0e3c_48c9_9997_46b62edb5072.slice/crio-922b42ea278d6ede529d5dc7b8ef10962cfb525ee32435768b0e4ea09d0f969b WatchSource:0}: Error finding container 922b42ea278d6ede529d5dc7b8ef10962cfb525ee32435768b0e4ea09d0f969b: Status 404 returned error can't find the container with id 922b42ea278d6ede529d5dc7b8ef10962cfb525ee32435768b0e4ea09d0f969b
Nov 03 07:24:19.945862 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.945834    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"]
Nov 03 07:24:19.947798 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.947766    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:24:19.947890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.947828    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:24:19.947890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:19.947859    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:24:20.782072 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:20.782028    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" event={"ID":"f9b225a1-0e3c-48c9-9997-46b62edb5072","Type":"ContainerStarted","Data":"520bbe46a01f9802f847b734ce2a28e6acfe1fe85e3639640ad5231b96e46f74"}
Nov 03 07:24:20.782296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:20.782091    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" event={"ID":"f9b225a1-0e3c-48c9-9997-46b62edb5072","Type":"ContainerStarted","Data":"922b42ea278d6ede529d5dc7b8ef10962cfb525ee32435768b0e4ea09d0f969b"}
Nov 03 07:24:20.977270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:20.977208    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" podStartSLOduration=2.9771922379999998 podStartE2EDuration="2.977192238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:24:20.973271041 +0000 UTC m=+1513.305640036" watchObservedRunningTime="2026-11-03 07:24:20.977192238 +0000 UTC m=+1513.309561232"
Nov 03 07:24:21.504890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:21.504860    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:21.506776 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:21.506748    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 07:24:21.506914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:21.506798    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" podUID="f9b225a1-0e3c-48c9-9997-46b62edb5072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 07:24:22.505917 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:22.505879    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 07:24:22.506443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:22.505927    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" podUID="f9b225a1-0e3c-48c9-9997-46b62edb5072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 07:24:23.506091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:23.506057    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 07:24:23.506514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:23.506111    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" podUID="f9b225a1-0e3c-48c9-9997-46b62edb5072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 07:24:24.505855 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:24.505819    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 07:24:24.506035 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:24.505879    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" podUID="f9b225a1-0e3c-48c9-9997-46b62edb5072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 07:24:25.506035 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:25.506000    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 07:24:25.506493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:25.506066    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" podUID="f9b225a1-0e3c-48c9-9997-46b62edb5072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 07:24:26.505861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:26.505819    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 07:24:26.506043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:26.505889    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" podUID="f9b225a1-0e3c-48c9-9997-46b62edb5072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 07:24:27.505839 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:27.505786    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 07:24:27.506259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:27.505867    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" podUID="f9b225a1-0e3c-48c9-9997-46b62edb5072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 07:24:28.510570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:28.510541    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:28.511051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:28.510910    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:28.511573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:28.511550    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb"
Nov 03 07:24:34.001664 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.001630    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"]
Nov 03 07:24:34.007351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.007319    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.118588 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.118553    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"]
Nov 03 07:24:34.128103 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.128071    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e33006-3dbb-4d05-a871-af997921a1c2-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.128320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.128115    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.128320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.128231    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e33006-3dbb-4d05-a871-af997921a1c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.128320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.128272    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e33006-3dbb-4d05-a871-af997921a1c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.128458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.128340    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.128458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.128395    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8qgf\" (UniqueName: \"kubernetes.io/projected/63e33006-3dbb-4d05-a871-af997921a1c2-kube-api-access-h8qgf\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.128612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.128589    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.128740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.128720    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.128827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.128813    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230031 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.229985    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8qgf\" (UniqueName: \"kubernetes.io/projected/63e33006-3dbb-4d05-a871-af997921a1c2-kube-api-access-h8qgf\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.230108    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.230264    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.230335    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.230407    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e33006-3dbb-4d05-a871-af997921a1c2-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.230447    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.230495    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e33006-3dbb-4d05-a871-af997921a1c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.230526    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.230574    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.230552    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e33006-3dbb-4d05-a871-af997921a1c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.230648    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.230836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.230666    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.231035 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.231015    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.231194 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.231175    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e33006-3dbb-4d05-a871-af997921a1c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.232981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.232958    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e33006-3dbb-4d05-a871-af997921a1c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.233195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.233159    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e33006-3dbb-4d05-a871-af997921a1c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.295053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.294988    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e33006-3dbb-4d05-a871-af997921a1c2-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.295238 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.295122    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8qgf\" (UniqueName: \"kubernetes.io/projected/63e33006-3dbb-4d05-a871-af997921a1c2-kube-api-access-h8qgf\") pod \"maas-default-gateway-openshift-default-58d898bd55-mrk4m\" (UID: \"63e33006-3dbb-4d05-a871-af997921a1c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.323463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.323424    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:34.759794 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:24:34.759760    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:24:34.775430 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:24:34.775393    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e33006_3dbb_4d05_a871_af997921a1c2.slice/crio-e3a06704c393742a2db7f89e319be07586c4d87ad3daeb620f1d43566777516e WatchSource:0}: Error finding container e3a06704c393742a2db7f89e319be07586c4d87ad3daeb620f1d43566777516e: Status 404 returned error can't find the container with id e3a06704c393742a2db7f89e319be07586c4d87ad3daeb620f1d43566777516e
Nov 03 07:24:34.778331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.778297    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:24:34.778454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.778381    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:24:34.778454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.778427    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:24:34.779019 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:34.779000    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"]
Nov 03 07:24:35.037207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:35.036741    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" event={"ID":"63e33006-3dbb-4d05-a871-af997921a1c2","Type":"ContainerStarted","Data":"ff0980422d8d1c2f7a464df8c0c49410ae745fa9b466e0db1c3d451c3179dac5"}
Nov 03 07:24:35.037207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:35.036809    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" event={"ID":"63e33006-3dbb-4d05-a871-af997921a1c2","Type":"ContainerStarted","Data":"e3a06704c393742a2db7f89e319be07586c4d87ad3daeb620f1d43566777516e"}
Nov 03 07:24:35.324413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:35.324371    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:35.325824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:35.325797    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 03 07:24:35.325964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:35.325841    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" podUID="63e33006-3dbb-4d05-a871-af997921a1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 03 07:24:36.324311 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:36.324275    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 03 07:24:36.324688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:36.324338    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" podUID="63e33006-3dbb-4d05-a871-af997921a1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 03 07:24:37.323821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:37.323781    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 03 07:24:37.324000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:37.323842    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" podUID="63e33006-3dbb-4d05-a871-af997921a1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 03 07:24:38.323975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:38.323939    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 03 07:24:38.324393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:38.323997    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" podUID="63e33006-3dbb-4d05-a871-af997921a1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 03 07:24:39.324099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:39.324065    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 03 07:24:39.324525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:39.324127    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" podUID="63e33006-3dbb-4d05-a871-af997921a1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 03 07:24:40.324624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:40.324593    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 03 07:24:40.325003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:40.324664    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" podUID="63e33006-3dbb-4d05-a871-af997921a1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 03 07:24:41.324434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:41.324398    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 03 07:24:41.324608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:41.324471    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" podUID="63e33006-3dbb-4d05-a871-af997921a1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 03 07:24:42.323879 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:42.323834    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 03 07:24:42.324415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:42.323923    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" podUID="63e33006-3dbb-4d05-a871-af997921a1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 03 07:24:43.646078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.646014    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" podStartSLOduration=10.645999091 podStartE2EDuration="10.645999091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:24:35.188344063 +0000 UTC m=+1527.520713058" watchObservedRunningTime="2026-11-03 07:24:43.645999091 +0000 UTC m=+1535.978368093"
Nov 03 07:24:43.648341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.648314    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"]
Nov 03 07:24:43.653160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.653121    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:43.715801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.715765    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"]
Nov 03 07:24:43.726593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.726556    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:24:43.726593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.726556    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:24:43.756603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.756548    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:24:43.756796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.756604    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:24:43.773696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.773658    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67db4bc0-66z55\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:43.773931 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.773907    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z554\" (UniqueName: \"kubernetes.io/projected/c4869b76-0860-49ea-a0f3-a452299be2bb-kube-api-access-4z554\") pod \"maas-api-revoke-keys-e2e-ait-67db4bc0-66z55\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:43.774179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.774157    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67db4bc0-66z55\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:43.824260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.824230    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:43.825080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.825055    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:43.826618 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.826598    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m"
Nov 03 07:24:43.875474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.875439    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z554\" (UniqueName: \"kubernetes.io/projected/c4869b76-0860-49ea-a0f3-a452299be2bb-kube-api-access-4z554\") pod \"maas-api-revoke-keys-e2e-ait-67db4bc0-66z55\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:43.875683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.875646    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67db4bc0-66z55\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:43.875879 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.875856    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67db4bc0-66z55\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:43.876532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.876506    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67db4bc0-66z55\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:43.879106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.879080    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67db4bc0-66z55\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:43.940012 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.939946    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z554\" (UniqueName: \"kubernetes.io/projected/c4869b76-0860-49ea-a0f3-a452299be2bb-kube-api-access-4z554\") pod \"maas-api-revoke-keys-e2e-ait-67db4bc0-66z55\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:43.970981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:43.970953    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:44.388917 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:24:44.388883    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4869b76_0860_49ea_a0f3_a452299be2bb.slice/crio-8cccc3b57c4325b5459d087a9bc8e6332b6bb9b34bf82e886a38530a40dc2755 WatchSource:0}: Error finding container 8cccc3b57c4325b5459d087a9bc8e6332b6bb9b34bf82e886a38530a40dc2755: Status 404 returned error can't find the container with id 8cccc3b57c4325b5459d087a9bc8e6332b6bb9b34bf82e886a38530a40dc2755
Nov 03 07:24:44.390832 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:44.390807    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"]
Nov 03 07:24:44.828403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:44.828287    4470 patch_prober.go:29] interesting pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": read tcp 10.134.0.2:44694->10.134.0.129:15021: read: connection reset by peer" start-of-body=
Nov 03 07:24:44.828403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:44.828343    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": read tcp 10.134.0.2:44694->10.134.0.129:15021: read: connection reset by peer"
Nov 03 07:24:45.301736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:45.301703    4470 generic.go:356] "Generic (PLEG): container finished" podID="c4869b76-0860-49ea-a0f3-a452299be2bb" containerID="cbc3e3b62cf654a47aa9e5624ba7e5225629b81344925928510b3ec61f8f0f96" exitCode=0
Nov 03 07:24:45.301875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:45.301786    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55" event={"ID":"c4869b76-0860-49ea-a0f3-a452299be2bb","Type":"ContainerDied","Data":"cbc3e3b62cf654a47aa9e5624ba7e5225629b81344925928510b3ec61f8f0f96"}
Nov 03 07:24:45.301875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:45.301835    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55" event={"ID":"c4869b76-0860-49ea-a0f3-a452299be2bb","Type":"ContainerStarted","Data":"8cccc3b57c4325b5459d087a9bc8e6332b6bb9b34bf82e886a38530a40dc2755"}
Nov 03 07:24:45.316491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:45.316461    4470 generic.go:356] "Generic (PLEG): container finished" podID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerID="2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e" exitCode=0
Nov 03 07:24:45.316624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:45.316530    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" event={"ID":"06dd1225-38d7-4297-9c65-42e0788fbe3a","Type":"ContainerDied","Data":"2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e"}
Nov 03 07:24:45.317042 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:45.317028    4470 scope.go:122] "RemoveContainer" containerID="2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e"
Nov 03 07:24:45.318428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:45.318401    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:24:45.318489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:45.318467    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:24:45.318530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:45.318501    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:24:46.327462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.327431    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" event={"ID":"06dd1225-38d7-4297-9c65-42e0788fbe3a","Type":"ContainerStarted","Data":"6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d"}
Nov 03 07:24:46.536737 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.536712    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:46.610386 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.610301    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:24:46.610605 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.610575    4470 patch_prober.go:29] interesting pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 07:24:46.610738 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.610630    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 07:24:46.631086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.631056    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-xks-ca\") pod \"c4869b76-0860-49ea-a0f3-a452299be2bb\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") "
Nov 03 07:24:46.631400 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.631386    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-openshift-ca\") pod \"c4869b76-0860-49ea-a0f3-a452299be2bb\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") "
Nov 03 07:24:46.631453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.631432    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4869b76-0860-49ea-a0f3-a452299be2bb-kube-api-access-4z554\" (UniqueName: \"kubernetes.io/projected/c4869b76-0860-49ea-a0f3-a452299be2bb-kube-api-access-4z554\") pod \"c4869b76-0860-49ea-a0f3-a452299be2bb\" (UID: \"c4869b76-0860-49ea-a0f3-a452299be2bb\") "
Nov 03 07:24:46.631743 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.631721    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-openshift-ca" pod "c4869b76-0860-49ea-a0f3-a452299be2bb" (UID: "c4869b76-0860-49ea-a0f3-a452299be2bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:24:46.632030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.632014    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:24:46.633271 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.633243    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-xks-ca" pod "c4869b76-0860-49ea-a0f3-a452299be2bb" (UID: "c4869b76-0860-49ea-a0f3-a452299be2bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:24:46.633668 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.633651    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4869b76-0860-49ea-a0f3-a452299be2bb-kube-api-access-4z554" pod "c4869b76-0860-49ea-a0f3-a452299be2bb" (UID: "c4869b76-0860-49ea-a0f3-a452299be2bb"). InnerVolumeSpecName "kube-api-access-4z554". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:24:46.733315 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.733274    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z554\" (UniqueName: \"kubernetes.io/projected/c4869b76-0860-49ea-a0f3-a452299be2bb-kube-api-access-4z554\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:24:46.733315 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:46.733309    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4869b76-0860-49ea-a0f3-a452299be2bb-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:24:47.336834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:47.336805    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"
Nov 03 07:24:47.336834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:47.336818    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55" event={"ID":"c4869b76-0860-49ea-a0f3-a452299be2bb","Type":"ContainerDied","Data":"8cccc3b57c4325b5459d087a9bc8e6332b6bb9b34bf82e886a38530a40dc2755"}
Nov 03 07:24:47.337412 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:47.336868    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cccc3b57c4325b5459d087a9bc8e6332b6bb9b34bf82e886a38530a40dc2755"
Nov 03 07:24:47.610168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:47.610056    4470 patch_prober.go:29] interesting pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 07:24:47.610371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:47.610117    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 07:24:48.610859 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.610823    4470 patch_prober.go:29] interesting pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 07:24:48.611282 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.610882    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 07:24:48.642351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.642318    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"]
Nov 03 07:24:48.643032 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.643018    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c4869b76-0860-49ea-a0f3-a452299be2bb" containerName="revoke-keys"
Nov 03 07:24:48.643085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.643033    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4869b76-0860-49ea-a0f3-a452299be2bb" containerName="revoke-keys"
Nov 03 07:24:48.643192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.643173    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4869b76-0860-49ea-a0f3-a452299be2bb" containerName="revoke-keys"
Nov 03 07:24:48.664416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.664389    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:48.702539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.702507    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"]
Nov 03 07:24:48.723350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.723321    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:24:48.723350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.723346    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:24:48.772937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.772902    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:24:48.773841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.773807    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:24:48.787079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.787045    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:48.787308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.787278    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:48.787526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.787509    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz94h\" (UniqueName: \"kubernetes.io/projected/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-kube-api-access-hz94h\") pod \"maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:48.888847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.888759    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz94h\" (UniqueName: \"kubernetes.io/projected/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-kube-api-access-hz94h\") pod \"maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:48.889025 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.888985    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:48.889090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.889021    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:48.889730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.889686    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:48.891584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.891561    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:48.970166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.969927    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz94h\" (UniqueName: \"kubernetes.io/projected/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-kube-api-access-hz94h\") pod \"maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:48.986759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:48.986724    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:49.412440 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:24:49.412410    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71356d65_aa35_4c4d_a3a8_2324fb1fa1bd.slice/crio-d8e73656eef795b134ef09700cfeeea6e34f90d8bd0353e4667a12837f6bb41f WatchSource:0}: Error finding container d8e73656eef795b134ef09700cfeeea6e34f90d8bd0353e4667a12837f6bb41f: Status 404 returned error can't find the container with id d8e73656eef795b134ef09700cfeeea6e34f90d8bd0353e4667a12837f6bb41f
Nov 03 07:24:49.413508 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:49.413486    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"]
Nov 03 07:24:49.610699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:49.610657    4470 patch_prober.go:29] interesting pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 07:24:49.610865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:49.610719    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 07:24:50.388090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:50.388054    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk" event={"ID":"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd","Type":"ContainerStarted","Data":"343123871b70f0202ed8214c50528569918b0e7a4f83d227e5c219e6d607c9c1"}
Nov 03 07:24:50.388555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:50.388108    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk" event={"ID":"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd","Type":"ContainerStarted","Data":"d8e73656eef795b134ef09700cfeeea6e34f90d8bd0353e4667a12837f6bb41f"}
Nov 03 07:24:50.572523 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:50.572462    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk" podStartSLOduration=2.5724429559999997 podStartE2EDuration="2.572442956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:24:50.511068498 +0000 UTC m=+1542.843437492" watchObservedRunningTime="2026-11-03 07:24:50.572442956 +0000 UTC m=+1542.904811955"
Nov 03 07:24:50.610182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:50.610127    4470 patch_prober.go:29] interesting pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 07:24:50.610373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:50.610199    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 07:24:51.398755 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.398714    4470 generic.go:356] "Generic (PLEG): container finished" podID="71356d65-aa35-4c4d-a3a8-2324fb1fa1bd" containerID="343123871b70f0202ed8214c50528569918b0e7a4f83d227e5c219e6d607c9c1" exitCode=0
Nov 03 07:24:51.399201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.398793    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk" event={"ID":"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd","Type":"ContainerDied","Data":"343123871b70f0202ed8214c50528569918b0e7a4f83d227e5c219e6d607c9c1"}
Nov 03 07:24:51.518060 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.518031    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"]
Nov 03 07:24:51.522714 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.522695    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:51.582190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.582158    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"]
Nov 03 07:24:51.610933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.610742    4470 patch_prober.go:29] interesting pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 07:24:51.610933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.610802    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 07:24:51.634740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.634706    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87b8bc5f-c5qmh\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:51.634929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.634752    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfwf\" (UniqueName: \"kubernetes.io/projected/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-kube-api-access-jwfwf\") pod \"maas-api-revoke-sub-87b8bc5f-c5qmh\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:51.634929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.634846    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87b8bc5f-c5qmh\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:51.736972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.736881    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87b8bc5f-c5qmh\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:51.736972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.736925    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfwf\" (UniqueName: \"kubernetes.io/projected/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-kube-api-access-jwfwf\") pod \"maas-api-revoke-sub-87b8bc5f-c5qmh\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:51.736972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.736966    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87b8bc5f-c5qmh\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:51.737578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.737555    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87b8bc5f-c5qmh\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:51.739667 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.739644    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87b8bc5f-c5qmh\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:51.804181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.804147    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfwf\" (UniqueName: \"kubernetes.io/projected/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-kube-api-access-jwfwf\") pod \"maas-api-revoke-sub-87b8bc5f-c5qmh\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:51.837057 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:51.837030    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:52.073469 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.073433    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"]
Nov 03 07:24:52.079273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.079245    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7ce3ae9-t8drs"]
Nov 03 07:24:52.220748 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:24:52.220720    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbeeaa3d_4c6c_4324_8aba_2585e7518b62.slice/crio-50c232ac5f3b48a6f16d11d1193244b64b1d29168374aa196c35201da0951f90 WatchSource:0}: Error finding container 50c232ac5f3b48a6f16d11d1193244b64b1d29168374aa196c35201da0951f90: Status 404 returned error can't find the container with id 50c232ac5f3b48a6f16d11d1193244b64b1d29168374aa196c35201da0951f90
Nov 03 07:24:52.223277 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.223256    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"]
Nov 03 07:24:52.408447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.408362    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh" event={"ID":"fbeeaa3d-4c6c-4324-8aba-2585e7518b62","Type":"ContainerStarted","Data":"50c232ac5f3b48a6f16d11d1193244b64b1d29168374aa196c35201da0951f90"}
Nov 03 07:24:52.556404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.556377    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:52.615527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.615497    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:24:52.615946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.615925    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:24:52.616831 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.616812    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:24:52.752340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.752309    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-kube-api-access-hz94h\" (UniqueName: \"kubernetes.io/projected/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-kube-api-access-hz94h\") pod \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") "
Nov 03 07:24:52.752481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.752390    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-openshift-ca\") pod \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") "
Nov 03 07:24:52.752550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.752516    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-xks-ca\") pod \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\" (UID: \"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd\") "
Nov 03 07:24:52.752796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.752752    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-openshift-ca" pod "71356d65-aa35-4c4d-a3a8-2324fb1fa1bd" (UID: "71356d65-aa35-4c4d-a3a8-2324fb1fa1bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:24:52.753460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.753437    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:24:52.754828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.754805    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-kube-api-access-hz94h" pod "71356d65-aa35-4c4d-a3a8-2324fb1fa1bd" (UID: "71356d65-aa35-4c4d-a3a8-2324fb1fa1bd"). InnerVolumeSpecName "kube-api-access-hz94h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:24:52.755248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.755226    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-xks-ca" pod "71356d65-aa35-4c4d-a3a8-2324fb1fa1bd" (UID: "71356d65-aa35-4c4d-a3a8-2324fb1fa1bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:24:52.855894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.855858    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz94h\" (UniqueName: \"kubernetes.io/projected/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-kube-api-access-hz94h\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:24:52.855894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:52.855897    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:24:53.418254 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:53.418175    4470 generic.go:356] "Generic (PLEG): container finished" podID="fbeeaa3d-4c6c-4324-8aba-2585e7518b62" containerID="db05ee991b656051438b13ad53f1d15c9393030b305058387725e78a357208b5" exitCode=0
Nov 03 07:24:53.418254 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:53.418230    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh" event={"ID":"fbeeaa3d-4c6c-4324-8aba-2585e7518b62","Type":"ContainerDied","Data":"db05ee991b656051438b13ad53f1d15c9393030b305058387725e78a357208b5"}
Nov 03 07:24:53.419829 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:53.419810    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"
Nov 03 07:24:53.419915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:53.419855    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk" event={"ID":"71356d65-aa35-4c4d-a3a8-2324fb1fa1bd","Type":"ContainerDied","Data":"d8e73656eef795b134ef09700cfeeea6e34f90d8bd0353e4667a12837f6bb41f"}
Nov 03 07:24:53.419915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:53.419898    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8e73656eef795b134ef09700cfeeea6e34f90d8bd0353e4667a12837f6bb41f"
Nov 03 07:24:53.807950 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:53.807870    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af623390-0e79-46d3-902e-15248baf6f6a" path="/var/lib/kubelet/pods/af623390-0e79-46d3-902e-15248baf6f6a/volumes"
Nov 03 07:24:54.592335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:54.592312    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:54.680927 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:54.680839    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-xks-ca\") pod \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") "
Nov 03 07:24:54.681077 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:54.681011    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-kube-api-access-jwfwf\" (UniqueName: \"kubernetes.io/projected/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-kube-api-access-jwfwf\") pod \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") "
Nov 03 07:24:54.681125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:54.681079    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-openshift-ca\") pod \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\" (UID: \"fbeeaa3d-4c6c-4324-8aba-2585e7518b62\") "
Nov 03 07:24:54.681468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:54.681443    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-openshift-ca" pod "fbeeaa3d-4c6c-4324-8aba-2585e7518b62" (UID: "fbeeaa3d-4c6c-4324-8aba-2585e7518b62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:24:54.682252 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:54.682222    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:24:54.683692 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:54.683646    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-kube-api-access-jwfwf" pod "fbeeaa3d-4c6c-4324-8aba-2585e7518b62" (UID: "fbeeaa3d-4c6c-4324-8aba-2585e7518b62"). InnerVolumeSpecName "kube-api-access-jwfwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:24:54.683692 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:54.683661    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-xks-ca" pod "fbeeaa3d-4c6c-4324-8aba-2585e7518b62" (UID: "fbeeaa3d-4c6c-4324-8aba-2585e7518b62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:24:54.783497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:54.783465    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:24:54.783497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:54.783493    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwfwf\" (UniqueName: \"kubernetes.io/projected/fbeeaa3d-4c6c-4324-8aba-2585e7518b62-kube-api-access-jwfwf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:24:55.196161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:55.194150    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"]
Nov 03 07:24:55.439484 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:55.439458    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"
Nov 03 07:24:55.439677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:55.439488    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh" event={"ID":"fbeeaa3d-4c6c-4324-8aba-2585e7518b62","Type":"ContainerDied","Data":"50c232ac5f3b48a6f16d11d1193244b64b1d29168374aa196c35201da0951f90"}
Nov 03 07:24:55.439677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:24:55.439564    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c232ac5f3b48a6f16d11d1193244b64b1d29168374aa196c35201da0951f90"
Nov 03 07:25:00.194842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:00.194799    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9" podUID="3d978992-ceae-4b80-9c92-519ec5d9775b" containerName="payload-processing" containerID="cri-o://8126a5c604fddc0e4e00836af44fb21b22771018a408b5439dcc106918591d02" gracePeriod=25
Nov 03 07:25:00.494583 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:00.494496    4470 generic.go:356] "Generic (PLEG): container finished" podID="3d978992-ceae-4b80-9c92-519ec5d9775b" containerID="8126a5c604fddc0e4e00836af44fb21b22771018a408b5439dcc106918591d02" exitCode=0
Nov 03 07:25:00.494841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:00.494817    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9" event={"ID":"3d978992-ceae-4b80-9c92-519ec5d9775b","Type":"ContainerDied","Data":"8126a5c604fddc0e4e00836af44fb21b22771018a408b5439dcc106918591d02"}
Nov 03 07:25:00.556175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:00.556108    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:25:00.744566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:00.743906    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d978992-ceae-4b80-9c92-519ec5d9775b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d978992-ceae-4b80-9c92-519ec5d9775b-plugins-config-volume\") pod \"3d978992-ceae-4b80-9c92-519ec5d9775b\" (UID: \"3d978992-ceae-4b80-9c92-519ec5d9775b\") "
Nov 03 07:25:00.744566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:00.744202    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d978992-ceae-4b80-9c92-519ec5d9775b-plugins-config-volume" pod "3d978992-ceae-4b80-9c92-519ec5d9775b" (UID: "3d978992-ceae-4b80-9c92-519ec5d9775b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:25:00.744566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:00.744206    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d978992-ceae-4b80-9c92-519ec5d9775b-kube-api-access-vw2zf\" (UniqueName: \"kubernetes.io/projected/3d978992-ceae-4b80-9c92-519ec5d9775b-kube-api-access-vw2zf\") pod \"3d978992-ceae-4b80-9c92-519ec5d9775b\" (UID: \"3d978992-ceae-4b80-9c92-519ec5d9775b\") "
Nov 03 07:25:00.745260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:00.745039    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d978992-ceae-4b80-9c92-519ec5d9775b-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:00.747828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:00.747803    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d978992-ceae-4b80-9c92-519ec5d9775b-kube-api-access-vw2zf" pod "3d978992-ceae-4b80-9c92-519ec5d9775b" (UID: "3d978992-ceae-4b80-9c92-519ec5d9775b"). InnerVolumeSpecName "kube-api-access-vw2zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:25:00.846846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:00.846813    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw2zf\" (UniqueName: \"kubernetes.io/projected/3d978992-ceae-4b80-9c92-519ec5d9775b-kube-api-access-vw2zf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:01.504602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:01.504575    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"
Nov 03 07:25:01.505121 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:01.504572    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9" event={"ID":"3d978992-ceae-4b80-9c92-519ec5d9775b","Type":"ContainerDied","Data":"8feb20ee3464991bbe76ebe023fefe1f08b408418abc8f0cde174d8f6f97aafa"}
Nov 03 07:25:01.505121 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:01.504714    4470 scope.go:122] "RemoveContainer" containerID="8126a5c604fddc0e4e00836af44fb21b22771018a408b5439dcc106918591d02"
Nov 03 07:25:01.693263 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:01.691503    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"]
Nov 03 07:25:01.771062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:01.771029    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9"]
Nov 03 07:25:01.815263 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:01.815226    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d978992-ceae-4b80-9c92-519ec5d9775b" path="/var/lib/kubelet/pods/3d978992-ceae-4b80-9c92-519ec5d9775b/volumes"
Nov 03 07:25:02.077362 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:02.077234    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"]
Nov 03 07:25:02.085510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:02.085483    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1952a2-zphwx"]
Nov 03 07:25:03.807489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:03.807455    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2582dd-7fc3-4c6a-aa5c-7e79388880e6" path="/var/lib/kubelet/pods/2a2582dd-7fc3-4c6a-aa5c-7e79388880e6/volumes"
Nov 03 07:25:04.759814 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:25:04.759784    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:25:19.736800 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.736698    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"]
Nov 03 07:25:19.737652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.737634    4470 cpu_manager.go:414] "containerMap: removing container" podUID="fbeeaa3d-4c6c-4324-8aba-2585e7518b62" containerName="revoke-keys"
Nov 03 07:25:19.737708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.737654    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbeeaa3d-4c6c-4324-8aba-2585e7518b62" containerName="revoke-keys"
Nov 03 07:25:19.737708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.737669    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3d978992-ceae-4b80-9c92-519ec5d9775b" containerName="payload-processing"
Nov 03 07:25:19.737708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.737675    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d978992-ceae-4b80-9c92-519ec5d9775b" containerName="payload-processing"
Nov 03 07:25:19.737708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.737681    4470 cpu_manager.go:414] "containerMap: removing container" podUID="71356d65-aa35-4c4d-a3a8-2324fb1fa1bd" containerName="revoke-keys"
Nov 03 07:25:19.737708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.737686    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71356d65-aa35-4c4d-a3a8-2324fb1fa1bd" containerName="revoke-keys"
Nov 03 07:25:19.737863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.737825    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbeeaa3d-4c6c-4324-8aba-2585e7518b62" containerName="revoke-keys"
Nov 03 07:25:19.737863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.737833    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d978992-ceae-4b80-9c92-519ec5d9775b" containerName="payload-processing"
Nov 03 07:25:19.737863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.737839    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="71356d65-aa35-4c4d-a3a8-2324fb1fa1bd" containerName="revoke-keys"
Nov 03 07:25:19.743357 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.743339    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:19.817187 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.817155    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"]
Nov 03 07:25:19.844128 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.844079    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:25:19.844509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.844485    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:25:19.844640 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.844543    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:25:19.852782 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.849613    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:25:19.858837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.858807    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mppfb\" (UniqueName: \"kubernetes.io/projected/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-kube-api-access-mppfb\") pod \"maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:19.858984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.858892    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:19.858984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.858961    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:19.961520 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.961492    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppfb\" (UniqueName: \"kubernetes.io/projected/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-kube-api-access-mppfb\") pod \"maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:19.961677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.961544    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:19.961677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.961587    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:19.962342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.962315    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:19.964182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:19.964165    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:20.026473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:20.026399    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mppfb\" (UniqueName: \"kubernetes.io/projected/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-kube-api-access-mppfb\") pod \"maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:20.057997 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:20.057963    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:20.060192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:20.060168    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"]
Nov 03 07:25:20.065722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:20.065695    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e636b1f-2vnpq"]
Nov 03 07:25:20.449195 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:25:20.449124    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c2a2df_4488_4cd8_8455_fbdaf5d79f95.slice/crio-3b69ef061ab7e426463bfb842613d310d3c511515d113f58bd3f1ff17532a140 WatchSource:0}: Error finding container 3b69ef061ab7e426463bfb842613d310d3c511515d113f58bd3f1ff17532a140: Status 404 returned error can't find the container with id 3b69ef061ab7e426463bfb842613d310d3c511515d113f58bd3f1ff17532a140
Nov 03 07:25:20.450939 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:20.450675    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"]
Nov 03 07:25:20.703777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:20.703693    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn" event={"ID":"67c2a2df-4488-4cd8-8455-fbdaf5d79f95","Type":"ContainerStarted","Data":"3b69ef061ab7e426463bfb842613d310d3c511515d113f58bd3f1ff17532a140"}
Nov 03 07:25:21.714927 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:21.714844    4470 generic.go:356] "Generic (PLEG): container finished" podID="67c2a2df-4488-4cd8-8455-fbdaf5d79f95" containerID="332c69d355fb1f703ce9d2f836642595a1fb9506b38b15091e8a8ba5cfe9b559" exitCode=0
Nov 03 07:25:21.715342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:21.714931    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn" event={"ID":"67c2a2df-4488-4cd8-8455-fbdaf5d79f95","Type":"ContainerDied","Data":"332c69d355fb1f703ce9d2f836642595a1fb9506b38b15091e8a8ba5cfe9b559"}
Nov 03 07:25:21.826540 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:21.826503    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793930bc-31b3-4c8d-9c4d-c272abfb8d96" path="/var/lib/kubelet/pods/793930bc-31b3-4c8d-9c4d-c272abfb8d96/volumes"
Nov 03 07:25:22.887753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:22.887730    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:23.008908 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.008166    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-xks-ca\") pod \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") "
Nov 03 07:25:23.008908 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.008329    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-kube-api-access-mppfb\" (UniqueName: \"kubernetes.io/projected/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-kube-api-access-mppfb\") pod \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") "
Nov 03 07:25:23.008908 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.008364    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-openshift-ca\") pod \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\" (UID: \"67c2a2df-4488-4cd8-8455-fbdaf5d79f95\") "
Nov 03 07:25:23.009532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.009505    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-openshift-ca" pod "67c2a2df-4488-4cd8-8455-fbdaf5d79f95" (UID: "67c2a2df-4488-4cd8-8455-fbdaf5d79f95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:25:23.012487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.012462    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-xks-ca" pod "67c2a2df-4488-4cd8-8455-fbdaf5d79f95" (UID: "67c2a2df-4488-4cd8-8455-fbdaf5d79f95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:25:23.012578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.012487    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-kube-api-access-mppfb" pod "67c2a2df-4488-4cd8-8455-fbdaf5d79f95" (UID: "67c2a2df-4488-4cd8-8455-fbdaf5d79f95"). InnerVolumeSpecName "kube-api-access-mppfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:25:23.110681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.110643    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:23.110681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.110676    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mppfb\" (UniqueName: \"kubernetes.io/projected/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-kube-api-access-mppfb\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:23.110884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.110691    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c2a2df-4488-4cd8-8455-fbdaf5d79f95-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:23.732928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.732898    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"
Nov 03 07:25:23.732928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.732924    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn" event={"ID":"67c2a2df-4488-4cd8-8455-fbdaf5d79f95","Type":"ContainerDied","Data":"3b69ef061ab7e426463bfb842613d310d3c511515d113f58bd3f1ff17532a140"}
Nov 03 07:25:23.733228 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:23.732967    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b69ef061ab7e426463bfb842613d310d3c511515d113f58bd3f1ff17532a140"
Nov 03 07:25:33.113236 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:33.113204    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"]
Nov 03 07:25:33.178559 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:33.175651    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67db4bc0-66z55"]
Nov 03 07:25:33.808593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:33.808556    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4869b76-0860-49ea-a0f3-a452299be2bb" path="/var/lib/kubelet/pods/c4869b76-0860-49ea-a0f3-a452299be2bb/volumes"
Nov 03 07:25:34.262596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:34.262561    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"]
Nov 03 07:25:34.267935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:34.267908    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-33a46b37-ht5nk"]
Nov 03 07:25:34.342797 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:34.342757    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"]
Nov 03 07:25:34.343151 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:34.343100    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy" containerID="cri-o://6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d" gracePeriod=30
Nov 03 07:25:34.767703 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:25:34.767663    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:25:35.812273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:35.812239    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71356d65-aa35-4c4d-a3a8-2324fb1fa1bd" path="/var/lib/kubelet/pods/71356d65-aa35-4c4d-a3a8-2324fb1fa1bd/volumes"
Nov 03 07:25:36.059962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:36.059928    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"]
Nov 03 07:25:36.110001 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:36.109927    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead06e14-8rhlj"]
Nov 03 07:25:37.626815 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:37.626777    4470 patch_prober.go:29] interesting pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:25:37.627368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:37.626859    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:25:37.809192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:37.809161    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43babcf9-6a25-49e8-a37b-5b4ced2e6ba0" path="/var/lib/kubelet/pods/43babcf9-6a25-49e8-a37b-5b4ced2e6ba0/volumes"
Nov 03 07:25:39.668443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.668419    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:25:39.775726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.775689    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-kube-api-access-rddkx\" (UniqueName: \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-kube-api-access-rddkx\") pod \"06dd1225-38d7-4297-9c65-42e0788fbe3a\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") "
Nov 03 07:25:39.775921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.775829    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-envoy\") pod \"06dd1225-38d7-4297-9c65-42e0788fbe3a\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") "
Nov 03 07:25:39.775921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.775879    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06dd1225-38d7-4297-9c65-42e0788fbe3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06dd1225-38d7-4297-9c65-42e0788fbe3a-istiod-ca-cert\") pod \"06dd1225-38d7-4297-9c65-42e0788fbe3a\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") "
Nov 03 07:25:39.776041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.775921    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-podinfo\") pod \"06dd1225-38d7-4297-9c65-42e0788fbe3a\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") "
Nov 03 07:25:39.776041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.776015    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-credential-socket\") pod \"06dd1225-38d7-4297-9c65-42e0788fbe3a\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") "
Nov 03 07:25:39.776190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.776173    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-certs\") pod \"06dd1225-38d7-4297-9c65-42e0788fbe3a\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") "
Nov 03 07:25:39.776305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.776287    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-data\") pod \"06dd1225-38d7-4297-9c65-42e0788fbe3a\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") "
Nov 03 07:25:39.776386 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.776314    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-credential-socket" pod "06dd1225-38d7-4297-9c65-42e0788fbe3a" (UID: "06dd1225-38d7-4297-9c65-42e0788fbe3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:25:39.776386 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.776329    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06dd1225-38d7-4297-9c65-42e0788fbe3a-istiod-ca-cert" pod "06dd1225-38d7-4297-9c65-42e0788fbe3a" (UID: "06dd1225-38d7-4297-9c65-42e0788fbe3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:25:39.776386 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.776348    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-socket\") pod \"06dd1225-38d7-4297-9c65-42e0788fbe3a\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") "
Nov 03 07:25:39.776539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.776439    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-token\" (UniqueName: \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-token\") pod \"06dd1225-38d7-4297-9c65-42e0788fbe3a\" (UID: \"06dd1225-38d7-4297-9c65-42e0788fbe3a\") "
Nov 03 07:25:39.776539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.776486    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-certs" pod "06dd1225-38d7-4297-9c65-42e0788fbe3a" (UID: "06dd1225-38d7-4297-9c65-42e0788fbe3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:25:39.776706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.776682    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-data" pod "06dd1225-38d7-4297-9c65-42e0788fbe3a" (UID: "06dd1225-38d7-4297-9c65-42e0788fbe3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:25:39.776766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.776710    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-socket" pod "06dd1225-38d7-4297-9c65-42e0788fbe3a" (UID: "06dd1225-38d7-4297-9c65-42e0788fbe3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:25:39.777206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.777182    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:39.777206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.777206    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:39.777389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.777215    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:39.777389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.777223    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06dd1225-38d7-4297-9c65-42e0788fbe3a-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:39.777389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.777233    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:39.778337 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.778311    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-kube-api-access-rddkx" pod "06dd1225-38d7-4297-9c65-42e0788fbe3a" (UID: "06dd1225-38d7-4297-9c65-42e0788fbe3a"). InnerVolumeSpecName "kube-api-access-rddkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:25:39.778743 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.778721    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-token" pod "06dd1225-38d7-4297-9c65-42e0788fbe3a" (UID: "06dd1225-38d7-4297-9c65-42e0788fbe3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:25:39.778822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.778801    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-envoy" pod "06dd1225-38d7-4297-9c65-42e0788fbe3a" (UID: "06dd1225-38d7-4297-9c65-42e0788fbe3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:25:39.778861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.778803    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-podinfo" pod "06dd1225-38d7-4297-9c65-42e0788fbe3a" (UID: "06dd1225-38d7-4297-9c65-42e0788fbe3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:25:39.878733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.878687    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:39.878733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.878726    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rddkx\" (UniqueName: \"kubernetes.io/projected/06dd1225-38d7-4297-9c65-42e0788fbe3a-kube-api-access-rddkx\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:39.878733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.878741    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:39.879026 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:39.878756    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06dd1225-38d7-4297-9c65-42e0788fbe3a-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:40.145307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.145278    4470 generic.go:356] "Generic (PLEG): container finished" podID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerID="6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d" exitCode=0
Nov 03 07:25:40.145496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.145332    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"
Nov 03 07:25:40.145496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.145351    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" event={"ID":"06dd1225-38d7-4297-9c65-42e0788fbe3a","Type":"ContainerDied","Data":"6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d"}
Nov 03 07:25:40.145496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.145412    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8" event={"ID":"06dd1225-38d7-4297-9c65-42e0788fbe3a","Type":"ContainerDied","Data":"56b16eec4557c336d93e9faf749c7754023eb19f4691a4ea25544d810c6a5369"}
Nov 03 07:25:40.145496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.145436    4470 scope.go:122] "RemoveContainer" containerID="6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d"
Nov 03 07:25:40.156764 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.156737    4470 scope.go:122] "RemoveContainer" containerID="2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e"
Nov 03 07:25:40.167300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.167280    4470 scope.go:122] "RemoveContainer" containerID="6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d"
Nov 03 07:25:40.167629 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:25:40.167600    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d\": container with ID starting with 6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d not found: ID does not exist" containerID="6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d"
Nov 03 07:25:40.167688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.167643    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d"} err="failed to get container status \"6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d\": rpc error: code = NotFound desc = could not find container \"6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d\": container with ID starting with 6fb9ab42b0120ddcf29976c8743eb0f586325f351a16dd09dd25d7aa8a30ee1d not found: ID does not exist"
Nov 03 07:25:40.167688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.167661    4470 scope.go:122] "RemoveContainer" containerID="2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e"
Nov 03 07:25:40.167933 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:25:40.167915    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e\": container with ID starting with 2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e not found: ID does not exist" containerID="2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e"
Nov 03 07:25:40.167978 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.167941    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e"} err="failed to get container status \"2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e\": rpc error: code = NotFound desc = could not find container \"2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e\": container with ID starting with 2a93e405cc59d2779bbb00399567413b05d64f22aa0671a338832a2df7a3b39e not found: ID does not exist"
Nov 03 07:25:40.357587 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.357546    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"]
Nov 03 07:25:40.378071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:40.378036    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8"]
Nov 03 07:25:41.812293 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:41.812258    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" path="/var/lib/kubelet/pods/06dd1225-38d7-4297-9c65-42e0788fbe3a/volumes"
Nov 03 07:25:48.930156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:48.928280    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"]
Nov 03 07:25:48.930156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:48.929392    4470 cpu_manager.go:414] "containerMap: removing container" podUID="67c2a2df-4488-4cd8-8455-fbdaf5d79f95" containerName="revoke-keys"
Nov 03 07:25:48.930156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:48.929412    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c2a2df-4488-4cd8-8455-fbdaf5d79f95" containerName="revoke-keys"
Nov 03 07:25:48.930156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:48.929438    4470 cpu_manager.go:414] "containerMap: removing container" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy"
Nov 03 07:25:48.930156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:48.929446    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy"
Nov 03 07:25:48.930156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:48.929502    4470 cpu_manager.go:414] "containerMap: removing container" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy"
Nov 03 07:25:48.930156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:48.929511    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy"
Nov 03 07:25:48.930156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:48.929672    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c2a2df-4488-4cd8-8455-fbdaf5d79f95" containerName="revoke-keys"
Nov 03 07:25:48.930156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:48.929687    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy"
Nov 03 07:25:48.930156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:48.929707    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="06dd1225-38d7-4297-9c65-42e0788fbe3a" containerName="istio-proxy"
Nov 03 07:25:49.096158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.091428    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:49.234124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.234036    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"]
Nov 03 07:25:49.238318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.238281    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:25:49.238636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.238621    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:25:49.238816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.238805    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:25:49.238957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.238946    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:25:49.337286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.337251    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9w87\" (UniqueName: \"kubernetes.io/projected/d8939c53-836d-4a76-9cec-5ec2bb3afc95-kube-api-access-x9w87\") pod \"maas-api-revoke-sub-18734a2e-nw6zq\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:49.337684 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.337659    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18734a2e-nw6zq\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:49.337754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.337699    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18734a2e-nw6zq\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:49.439180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.439112    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18734a2e-nw6zq\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:49.439374 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.439201    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18734a2e-nw6zq\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:49.439441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.439372    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9w87\" (UniqueName: \"kubernetes.io/projected/d8939c53-836d-4a76-9cec-5ec2bb3afc95-kube-api-access-x9w87\") pod \"maas-api-revoke-sub-18734a2e-nw6zq\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:49.439779 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.439755    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18734a2e-nw6zq\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:49.441747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.441724    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18734a2e-nw6zq\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:49.528265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.528101    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9w87\" (UniqueName: \"kubernetes.io/projected/d8939c53-836d-4a76-9cec-5ec2bb3afc95-kube-api-access-x9w87\") pod \"maas-api-revoke-sub-18734a2e-nw6zq\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:49.628317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:49.628289    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:50.323876 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:25:50.323845    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8939c53_836d_4a76_9cec_5ec2bb3afc95.slice/crio-c3ad2fb1701f2317b7a553c6cfc67ddb194d3a3c151ee95c9771dc603b8b36b7 WatchSource:0}: Error finding container c3ad2fb1701f2317b7a553c6cfc67ddb194d3a3c151ee95c9771dc603b8b36b7: Status 404 returned error can't find the container with id c3ad2fb1701f2317b7a553c6cfc67ddb194d3a3c151ee95c9771dc603b8b36b7
Nov 03 07:25:50.333519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:50.333495    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"]
Nov 03 07:25:50.336570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:50.336533    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq" event={"ID":"d8939c53-836d-4a76-9cec-5ec2bb3afc95","Type":"ContainerStarted","Data":"c3ad2fb1701f2317b7a553c6cfc67ddb194d3a3c151ee95c9771dc603b8b36b7"}
Nov 03 07:25:51.346217 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:51.346186    4470 generic.go:356] "Generic (PLEG): container finished" podID="d8939c53-836d-4a76-9cec-5ec2bb3afc95" containerID="f31c61f8a669e6af6aabde545c97a88ec2fed180796dae43976dc20f22ddd79c" exitCode=0
Nov 03 07:25:51.346596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:51.346248    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq" event={"ID":"d8939c53-836d-4a76-9cec-5ec2bb3afc95","Type":"ContainerDied","Data":"f31c61f8a669e6af6aabde545c97a88ec2fed180796dae43976dc20f22ddd79c"}
Nov 03 07:25:52.536681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:52.536647    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:52.602744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:52.602709    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-openshift-ca\") pod \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") "
Nov 03 07:25:52.602922 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:52.602798    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8939c53-836d-4a76-9cec-5ec2bb3afc95-kube-api-access-x9w87\" (UniqueName: \"kubernetes.io/projected/d8939c53-836d-4a76-9cec-5ec2bb3afc95-kube-api-access-x9w87\") pod \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") "
Nov 03 07:25:52.603010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:52.602927    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-xks-ca\") pod \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\" (UID: \"d8939c53-836d-4a76-9cec-5ec2bb3afc95\") "
Nov 03 07:25:52.603106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:52.603082    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-openshift-ca" pod "d8939c53-836d-4a76-9cec-5ec2bb3afc95" (UID: "d8939c53-836d-4a76-9cec-5ec2bb3afc95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:25:52.603493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:52.603461    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:52.605120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:52.605083    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-xks-ca" pod "d8939c53-836d-4a76-9cec-5ec2bb3afc95" (UID: "d8939c53-836d-4a76-9cec-5ec2bb3afc95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:25:52.605232 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:52.605212    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8939c53-836d-4a76-9cec-5ec2bb3afc95-kube-api-access-x9w87" pod "d8939c53-836d-4a76-9cec-5ec2bb3afc95" (UID: "d8939c53-836d-4a76-9cec-5ec2bb3afc95"). InnerVolumeSpecName "kube-api-access-x9w87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:25:52.705061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:52.705018    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9w87\" (UniqueName: \"kubernetes.io/projected/d8939c53-836d-4a76-9cec-5ec2bb3afc95-kube-api-access-x9w87\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:52.705061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:52.705062    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8939c53-836d-4a76-9cec-5ec2bb3afc95-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:25:53.365593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:53.365559    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"
Nov 03 07:25:53.365768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:53.365616    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq" event={"ID":"d8939c53-836d-4a76-9cec-5ec2bb3afc95","Type":"ContainerDied","Data":"c3ad2fb1701f2317b7a553c6cfc67ddb194d3a3c151ee95c9771dc603b8b36b7"}
Nov 03 07:25:53.365768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:53.365668    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3ad2fb1701f2317b7a553c6cfc67ddb194d3a3c151ee95c9771dc603b8b36b7"
Nov 03 07:25:54.075095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:54.075063    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"]
Nov 03 07:25:54.125533 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:54.125503    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa215abf-l4tqn"]
Nov 03 07:25:55.809383 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:25:55.809349    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="362d8b24-79e2-4289-8e58-71942688b8ea" path="/var/lib/kubelet/pods/362d8b24-79e2-4289-8e58-71942688b8ea/volumes"
Nov 03 07:26:02.061740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:02.061707    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"]
Nov 03 07:26:02.112201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:02.112168    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f64454-xtrg6"]
Nov 03 07:26:02.620526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:02.620492    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"]
Nov 03 07:26:02.670809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:02.670778    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5c169f6a-qzcfn"]
Nov 03 07:26:03.807941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:03.807907    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c2a2df-4488-4cd8-8455-fbdaf5d79f95" path="/var/lib/kubelet/pods/67c2a2df-4488-4cd8-8455-fbdaf5d79f95/volumes"
Nov 03 07:26:03.808624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:03.808607    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed08894-3603-4c93-bba3-ebc17ae888cc" path="/var/lib/kubelet/pods/aed08894-3603-4c93-bba3-ebc17ae888cc/volumes"
Nov 03 07:26:04.761871 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:26:04.761828    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:26:12.749871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:12.749837    4470 scope.go:122] "RemoveContainer" containerID="3abaa68d2a9bf401deec139d33d5c91fad9dc81dc62f44d02b530c4d384a3e8d"
Nov 03 07:26:12.760934 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:12.760915    4470 scope.go:122] "RemoveContainer" containerID="1350e6e23cbe9d1bd5a05b8a090f2f17fd919e8645b603770a39a276593e24cf"
Nov 03 07:26:17.060391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:17.060355    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"]
Nov 03 07:26:17.111454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:17.111422    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db28b20-qpd2h"]
Nov 03 07:26:17.808589 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:17.808548    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a73d109-b726-4ba4-97d3-bcb4c37e0f28" path="/var/lib/kubelet/pods/8a73d109-b726-4ba4-97d3-bcb4c37e0f28/volumes"
Nov 03 07:26:18.062601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:18.062575    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"]
Nov 03 07:26:18.114327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:18.114283    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a342e23f-djsxf"]
Nov 03 07:26:19.808591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:19.808556    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5979d46b-5f6e-4a03-aabb-631774d784f7" path="/var/lib/kubelet/pods/5979d46b-5f6e-4a03-aabb-631774d784f7/volumes"
Nov 03 07:26:34.376373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.376318    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"]
Nov 03 07:26:34.377087 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.377068    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d8939c53-836d-4a76-9cec-5ec2bb3afc95" containerName="revoke-keys"
Nov 03 07:26:34.377087 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.377086    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8939c53-836d-4a76-9cec-5ec2bb3afc95" containerName="revoke-keys"
Nov 03 07:26:34.377269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.377214    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8939c53-836d-4a76-9cec-5ec2bb3afc95" containerName="revoke-keys"
Nov 03 07:26:34.381011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.380995    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:26:34.443774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.443730    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzxb7\" (UniqueName: \"kubernetes.io/projected/81e93120-dd39-49a0-8b5c-1b394ac78cbb-kube-api-access-lzxb7\") pod \"payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl\" (UID: \"81e93120-dd39-49a0-8b5c-1b394ac78cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:26:34.444596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.444568    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e93120-dd39-49a0-8b5c-1b394ac78cbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl\" (UID: \"81e93120-dd39-49a0-8b5c-1b394ac78cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:26:34.457500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.457452    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"]
Nov 03 07:26:34.496203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.496076    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-893d83\""
Nov 03 07:26:34.496671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.496634    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-893d83-dockercfg-mbgzj\""
Nov 03 07:26:34.511465 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.511433    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"]
Nov 03 07:26:34.565750 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.565712    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e93120-dd39-49a0-8b5c-1b394ac78cbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl\" (UID: \"81e93120-dd39-49a0-8b5c-1b394ac78cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:26:34.571152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.571100    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzxb7\" (UniqueName: \"kubernetes.io/projected/81e93120-dd39-49a0-8b5c-1b394ac78cbb-kube-api-access-lzxb7\") pod \"payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl\" (UID: \"81e93120-dd39-49a0-8b5c-1b394ac78cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:26:34.572457 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.572431    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e93120-dd39-49a0-8b5c-1b394ac78cbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl\" (UID: \"81e93120-dd39-49a0-8b5c-1b394ac78cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:26:34.587408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.587376    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:26:34.611274 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.611238    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"]
Nov 03 07:26:34.641336 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.638564    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzxb7\" (UniqueName: \"kubernetes.io/projected/81e93120-dd39-49a0-8b5c-1b394ac78cbb-kube-api-access-lzxb7\") pod \"payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl\" (UID: \"81e93120-dd39-49a0-8b5c-1b394ac78cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:26:34.672896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.672860    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4hkl\" (UniqueName: \"kubernetes.io/projected/3581dd25-7393-4efb-989e-1c44e94263e5-kube-api-access-k4hkl\") pod \"payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv\" (UID: \"3581dd25-7393-4efb-989e-1c44e94263e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:26:34.673079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.673059    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3581dd25-7393-4efb-989e-1c44e94263e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv\" (UID: \"3581dd25-7393-4efb-989e-1c44e94263e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:26:34.697475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.697437    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:26:34.770216 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:26:34.770175    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:26:34.774754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.774578    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4hkl\" (UniqueName: \"kubernetes.io/projected/3581dd25-7393-4efb-989e-1c44e94263e5-kube-api-access-k4hkl\") pod \"payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv\" (UID: \"3581dd25-7393-4efb-989e-1c44e94263e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:26:34.774754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.774719    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3581dd25-7393-4efb-989e-1c44e94263e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv\" (UID: \"3581dd25-7393-4efb-989e-1c44e94263e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:26:34.775737 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.775713    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3581dd25-7393-4efb-989e-1c44e94263e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv\" (UID: \"3581dd25-7393-4efb-989e-1c44e94263e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:26:34.873416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.865878    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4hkl\" (UniqueName: \"kubernetes.io/projected/3581dd25-7393-4efb-989e-1c44e94263e5-kube-api-access-k4hkl\") pod \"payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv\" (UID: \"3581dd25-7393-4efb-989e-1c44e94263e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:26:34.924763 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:34.924663    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:26:35.109718 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:26:35.109276    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e93120_dd39_49a0_8b5c_1b394ac78cbb.slice/crio-bda859be07331038f9365702d76dac9ce6ce1a14cb7fc97e343974824410eed9 WatchSource:0}: Error finding container bda859be07331038f9365702d76dac9ce6ce1a14cb7fc97e343974824410eed9: Status 404 returned error can't find the container with id bda859be07331038f9365702d76dac9ce6ce1a14cb7fc97e343974824410eed9
Nov 03 07:26:35.111179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:35.110757    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"]
Nov 03 07:26:35.376761 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:26:35.376727    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3581dd25_7393_4efb_989e_1c44e94263e5.slice/crio-9d8d243deee730c49c0c6e008a0b52c6a2ea862b2f22cf41cc2e6dcbae6a6b52 WatchSource:0}: Error finding container 9d8d243deee730c49c0c6e008a0b52c6a2ea862b2f22cf41cc2e6dcbae6a6b52: Status 404 returned error can't find the container with id 9d8d243deee730c49c0c6e008a0b52c6a2ea862b2f22cf41cc2e6dcbae6a6b52
Nov 03 07:26:35.378181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:35.378108    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"]
Nov 03 07:26:35.937815 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:35.937778    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl" event={"ID":"81e93120-dd39-49a0-8b5c-1b394ac78cbb","Type":"ContainerStarted","Data":"bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937"}
Nov 03 07:26:35.938110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:35.937840    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl" event={"ID":"81e93120-dd39-49a0-8b5c-1b394ac78cbb","Type":"ContainerStarted","Data":"bda859be07331038f9365702d76dac9ce6ce1a14cb7fc97e343974824410eed9"}
Nov 03 07:26:35.938110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:35.937871    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:26:35.939764 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:35.939733    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv" event={"ID":"3581dd25-7393-4efb-989e-1c44e94263e5","Type":"ContainerStarted","Data":"c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6"}
Nov 03 07:26:35.939883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:35.939780    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv" event={"ID":"3581dd25-7393-4efb-989e-1c44e94263e5","Type":"ContainerStarted","Data":"9d8d243deee730c49c0c6e008a0b52c6a2ea862b2f22cf41cc2e6dcbae6a6b52"}
Nov 03 07:26:35.939883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:35.939801    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:26:36.060108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:36.060061    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl" podStartSLOduration=2.060045735 podStartE2EDuration="2.060045735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:26:36.057687582 +0000 UTC m=+1648.390056588" watchObservedRunningTime="2026-11-03 07:26:36.060045735 +0000 UTC m=+1648.392414728"
Nov 03 07:26:36.201720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:36.196526    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv" podStartSLOduration=2.196505305 podStartE2EDuration="2.196505305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:26:36.183449575 +0000 UTC m=+1648.515818569" watchObservedRunningTime="2026-11-03 07:26:36.196505305 +0000 UTC m=+1648.528874301"
Nov 03 07:26:40.058268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:40.058234    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"]
Nov 03 07:26:40.095370 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:40.095343    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ddb302-x9qww"]
Nov 03 07:26:41.094039 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:41.094011    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"]
Nov 03 07:26:41.099926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:41.099899    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bba0527-r8ttm"]
Nov 03 07:26:41.807429 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:41.807394    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85abeaec-c2dd-43ac-b037-76e1e9a735f8" path="/var/lib/kubelet/pods/85abeaec-c2dd-43ac-b037-76e1e9a735f8/volumes"
Nov 03 07:26:41.807957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:41.807932    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff624b96-4bef-4485-83a7-5ee92bd594a9" path="/var/lib/kubelet/pods/ff624b96-4bef-4485-83a7-5ee92bd594a9/volumes"
Nov 03 07:26:43.218082 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.218046    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"]
Nov 03 07:26:43.223847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.223826    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:26:43.235869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.235848    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f3adb8-b332-4a67-b345-561ed0da7c99-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg\" (UID: \"f3f3adb8-b332-4a67-b345-561ed0da7c99\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:26:43.235975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.235909    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxvrm\" (UniqueName: \"kubernetes.io/projected/f3f3adb8-b332-4a67-b345-561ed0da7c99-kube-api-access-vxvrm\") pod \"payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg\" (UID: \"f3f3adb8-b332-4a67-b345-561ed0da7c99\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:26:43.281023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.280992    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"]
Nov 03 07:26:43.327450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.327420    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-399d09e5-dockercfg-r2pps\""
Nov 03 07:26:43.327710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.327420    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-399d09e5\""
Nov 03 07:26:43.338200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.338105    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f3adb8-b332-4a67-b345-561ed0da7c99-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg\" (UID: \"f3f3adb8-b332-4a67-b345-561ed0da7c99\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:26:43.338365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.338229    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxvrm\" (UniqueName: \"kubernetes.io/projected/f3f3adb8-b332-4a67-b345-561ed0da7c99-kube-api-access-vxvrm\") pod \"payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg\" (UID: \"f3f3adb8-b332-4a67-b345-561ed0da7c99\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:26:43.340454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.339281    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f3adb8-b332-4a67-b345-561ed0da7c99-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg\" (UID: \"f3f3adb8-b332-4a67-b345-561ed0da7c99\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:26:43.474270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.473599    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxvrm\" (UniqueName: \"kubernetes.io/projected/f3f3adb8-b332-4a67-b345-561ed0da7c99-kube-api-access-vxvrm\") pod \"payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg\" (UID: \"f3f3adb8-b332-4a67-b345-561ed0da7c99\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:26:43.493681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.493652    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"]
Nov 03 07:26:43.531160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.527121    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:26:43.531160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.527478    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"]
Nov 03 07:26:43.547163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.541178    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:26:43.549028 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.547800    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5w4\" (UniqueName: \"kubernetes.io/projected/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-kube-api-access-zg5w4\") pod \"payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh\" (UID: \"afd8a6cd-4714-4d4a-ab14-55e8e0145b45\") " pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:26:43.549323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.549300    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-plugins-config-volume\") pod \"payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh\" (UID: \"afd8a6cd-4714-4d4a-ab14-55e8e0145b45\") " pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:26:43.656224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.656181    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5w4\" (UniqueName: \"kubernetes.io/projected/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-kube-api-access-zg5w4\") pod \"payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh\" (UID: \"afd8a6cd-4714-4d4a-ab14-55e8e0145b45\") " pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:26:43.656533 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.656507    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-plugins-config-volume\") pod \"payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh\" (UID: \"afd8a6cd-4714-4d4a-ab14-55e8e0145b45\") " pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:26:43.657415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.657389    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-plugins-config-volume\") pod \"payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh\" (UID: \"afd8a6cd-4714-4d4a-ab14-55e8e0145b45\") " pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:26:43.721101 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.721064    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5w4\" (UniqueName: \"kubernetes.io/projected/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-kube-api-access-zg5w4\") pod \"payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh\" (UID: \"afd8a6cd-4714-4d4a-ab14-55e8e0145b45\") " pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:26:43.870239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.870195    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:26:43.975092 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:26:43.975063    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f3adb8_b332_4a67_b345_561ed0da7c99.slice/crio-4d1c57fc99d11d70f16244eab7bba8385e79738a9932d2581fb5e43319430fb6 WatchSource:0}: Error finding container 4d1c57fc99d11d70f16244eab7bba8385e79738a9932d2581fb5e43319430fb6: Status 404 returned error can't find the container with id 4d1c57fc99d11d70f16244eab7bba8385e79738a9932d2581fb5e43319430fb6
Nov 03 07:26:43.975248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:43.975217    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"]
Nov 03 07:26:44.044645 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:44.044612    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg" event={"ID":"f3f3adb8-b332-4a67-b345-561ed0da7c99","Type":"ContainerStarted","Data":"4d1c57fc99d11d70f16244eab7bba8385e79738a9932d2581fb5e43319430fb6"}
Nov 03 07:26:44.058911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:44.058880    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"]
Nov 03 07:26:44.063691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:44.063664    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4d0691-cj2bf"]
Nov 03 07:26:44.256935 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:26:44.256903    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd8a6cd_4714_4d4a_ab14_55e8e0145b45.slice/crio-fcbcd867d2d6c4df14539dc5a41fac692198270831ce66d91c2d72ade8d13d6f WatchSource:0}: Error finding container fcbcd867d2d6c4df14539dc5a41fac692198270831ce66d91c2d72ade8d13d6f: Status 404 returned error can't find the container with id fcbcd867d2d6c4df14539dc5a41fac692198270831ce66d91c2d72ade8d13d6f
Nov 03 07:26:44.257321 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:44.257299    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"]
Nov 03 07:26:45.054084 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:45.054048    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg" event={"ID":"f3f3adb8-b332-4a67-b345-561ed0da7c99","Type":"ContainerStarted","Data":"15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b"}
Nov 03 07:26:45.054298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:45.054105    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:26:45.055895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:45.055871    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh" event={"ID":"afd8a6cd-4714-4d4a-ab14-55e8e0145b45","Type":"ContainerStarted","Data":"7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056"}
Nov 03 07:26:45.056017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:45.055910    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh" event={"ID":"afd8a6cd-4714-4d4a-ab14-55e8e0145b45","Type":"ContainerStarted","Data":"fcbcd867d2d6c4df14539dc5a41fac692198270831ce66d91c2d72ade8d13d6f"}
Nov 03 07:26:45.056017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:45.055931    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:26:45.175543 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:45.175483    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg" podStartSLOduration=2.17546873 podStartE2EDuration="2.17546873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:26:45.174408045 +0000 UTC m=+1657.506777040" watchObservedRunningTime="2026-11-03 07:26:45.17546873 +0000 UTC m=+1657.507837724"
Nov 03 07:26:45.328155 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:45.324696    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh" podStartSLOduration=2.32467691 podStartE2EDuration="2.32467691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:26:45.322392658 +0000 UTC m=+1657.654761653" watchObservedRunningTime="2026-11-03 07:26:45.32467691 +0000 UTC m=+1657.657045907"
Nov 03 07:26:45.808876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:45.808786    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d793c2d8-9346-4870-bdb2-2c93af03d30b" path="/var/lib/kubelet/pods/d793c2d8-9346-4870-bdb2-2c93af03d30b/volumes"
Nov 03 07:26:46.955153 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:46.955110    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:26:46.955651 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:46.955200    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:26:52.062915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:52.062827    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"]
Nov 03 07:26:52.113693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:52.113659    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb71197-w69pz"]
Nov 03 07:26:53.808907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:53.808865    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1053cc-fd90-4e3e-bebb-b877635991b1" path="/var/lib/kubelet/pods/9e1053cc-fd90-4e3e-bebb-b877635991b1/volumes"
Nov 03 07:26:56.066678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:56.066651    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:26:56.067070 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:56.066713    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:26:57.062880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:57.062852    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"]
Nov 03 07:26:57.113214 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:57.113183    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d980b3-r79x9"]
Nov 03 07:26:57.809427 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:26:57.809391    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ad0537-60f7-4845-921d-ef80d3e2e138" path="/var/lib/kubelet/pods/77ad0537-60f7-4845-921d-ef80d3e2e138/volumes"
Nov 03 07:27:02.806150 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.806096    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"]
Nov 03 07:27:02.812045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.812019    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.863212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.861695    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.863212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.861742    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.863212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.861799    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.863212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.861828    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.863212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.862037    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.863212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.862102    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.863212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.862161    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.864499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.864184    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkhg2\" (UniqueName: \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-kube-api-access-dkhg2\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.864499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.864237    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.875391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.874780    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-295fba7c-openshift-default-dockercfg-6kjds\""
Nov 03 07:27:02.950851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.936890    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"]
Nov 03 07:27:02.966652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.965522    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.966652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.965606    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.966652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.965649    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.966652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.965734    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkhg2\" (UniqueName: \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-kube-api-access-dkhg2\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.966652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.965770    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.966652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.965830    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.966652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.965855    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.966652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.965916    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.966652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.965970    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.970099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.970072    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.970975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.970763    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.972714 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.972385    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.972844 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.972666    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:02.973110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:02.973092    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:03.001896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:03.001827    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:03.013030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:03.012966    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:03.038470 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:03.038428    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:03.041968 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:03.040741    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkhg2\" (UniqueName: \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-kube-api-access-dkhg2\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:03.171184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:03.163453    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:03.668726 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:27:03.668694    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92f7a7c_240f_4bb9_a1e3_2fe9f653c0ba.slice/crio-2faaee200de29c952a2efb722cccd78c7ca3511d686c0f86e3db1ed8f05573e6 WatchSource:0}: Error finding container 2faaee200de29c952a2efb722cccd78c7ca3511d686c0f86e3db1ed8f05573e6: Status 404 returned error can't find the container with id 2faaee200de29c952a2efb722cccd78c7ca3511d686c0f86e3db1ed8f05573e6
Nov 03 07:27:03.669991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:03.669971    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"]
Nov 03 07:27:03.671445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:03.671412    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:27:03.671522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:03.671477    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:27:03.671522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:03.671504    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:27:04.060378 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:04.060306    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"]
Nov 03 07:27:04.069007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:04.068973    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a923f3-fdhx8"]
Nov 03 07:27:04.265590 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:04.265555    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" event={"ID":"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba","Type":"ContainerStarted","Data":"4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f"}
Nov 03 07:27:04.265803 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:04.265613    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" event={"ID":"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba","Type":"ContainerStarted","Data":"2faaee200de29c952a2efb722cccd78c7ca3511d686c0f86e3db1ed8f05573e6"}
Nov 03 07:27:04.404512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:04.404455    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" podStartSLOduration=2.404440868 podStartE2EDuration="2.404440868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:27:04.402320146 +0000 UTC m=+1676.734689146" watchObservedRunningTime="2026-11-03 07:27:04.404440868 +0000 UTC m=+1676.736809925"
Nov 03 07:27:04.762652 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:27:04.762575    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:27:05.164466 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:05.164435    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:05.169609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:05.169586    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:05.274204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:05.274173    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:05.275384 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:05.275364    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:27:05.808443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:05.808406    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c6a0a8-a112-4ab6-8066-65ce2701d6b6" path="/var/lib/kubelet/pods/83c6a0a8-a112-4ab6-8066-65ce2701d6b6/volumes"
Nov 03 07:27:06.058358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:06.058327    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"]
Nov 03 07:27:06.109277 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:06.109246    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52341cc7-pgvxz"]
Nov 03 07:27:07.810113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:07.810078    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57750d51-6e2b-4427-b9fa-b00346cf28a8" path="/var/lib/kubelet/pods/57750d51-6e2b-4427-b9fa-b00346cf28a8/volumes"
Nov 03 07:27:08.064460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:08.064430    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"]
Nov 03 07:27:08.114600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:08.114567    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06450ed2-snfq2"]
Nov 03 07:27:08.119111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:08.119090    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"]
Nov 03 07:27:08.122914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:08.122892    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a58829d4-ldkkq"]
Nov 03 07:27:09.808227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:09.808194    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c392bc-814d-4983-b3b0-ce8c23f8ac95" path="/var/lib/kubelet/pods/54c392bc-814d-4983-b3b0-ce8c23f8ac95/volumes"
Nov 03 07:27:09.808709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:09.808676    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b7fed8-56a7-473d-b5d1-a2848fd3748d" path="/var/lib/kubelet/pods/67b7fed8-56a7-473d-b5d1-a2848fd3748d/volumes"
Nov 03 07:27:12.839020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:12.838955    4470 scope.go:122] "RemoveContainer" containerID="4d4b500dd8707211c72f3a78141f6ba9f396b815e0b342c6c17a6e45f0f65dab"
Nov 03 07:27:12.873869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:12.873846    4470 scope.go:122] "RemoveContainer" containerID="903dd8b9b1989fbce5602683cea32547325794e314de453aded763b1ffbc12e1"
Nov 03 07:27:12.886305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:12.886276    4470 scope.go:122] "RemoveContainer" containerID="1c5e74e74e7a69bc36a98ebddbf6fb8189d7801fb4a45965618c567a280adb5f"
Nov 03 07:27:12.896648 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:12.896625    4470 scope.go:122] "RemoveContainer" containerID="bcdf9441ffe3784476739d7db5e1d4b7a36f9b6778215a626f21f1f37b972460"
Nov 03 07:27:13.058331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.058297    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"]
Nov 03 07:27:13.108192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.108094    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4edb5170-xlhgq"]
Nov 03 07:27:13.368005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.367930    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"]
Nov 03 07:27:13.375657 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.375632    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:27:13.427181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.427109    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"]
Nov 03 07:27:13.453477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.453426    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-295fba7c\""
Nov 03 07:27:13.453719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.453696    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-295fba7c-dockercfg-7n7qf\""
Nov 03 07:27:13.565529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.565489    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpqrs\" (UniqueName: \"kubernetes.io/projected/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-kube-api-access-kpqrs\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt\" (UID: \"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:27:13.565746 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.565693    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt\" (UID: \"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:27:13.667582 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.667488    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpqrs\" (UniqueName: \"kubernetes.io/projected/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-kube-api-access-kpqrs\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt\" (UID: \"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:27:13.667773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.667629    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt\" (UID: \"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:27:13.668380 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.668361    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt\" (UID: \"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:27:13.731973 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.731942    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpqrs\" (UniqueName: \"kubernetes.io/projected/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-kube-api-access-kpqrs\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt\" (UID: \"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:27:13.812159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.809987    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e66a022-ecb8-475d-b2c3-06a35e0e0da8" path="/var/lib/kubelet/pods/2e66a022-ecb8-475d-b2c3-06a35e0e0da8/volumes"
Nov 03 07:27:13.992408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:13.992325    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:27:14.064595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:14.064528    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"]
Nov 03 07:27:14.072691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:14.072662    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ffd35b-ttfnb"]
Nov 03 07:27:14.386158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:14.384792    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"]
Nov 03 07:27:14.388920 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:27:14.388895    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4062b59b_02fe_4ed1_b3a8_4b20c296c6ee.slice/crio-48d795b1e5f0513050342c747b329fabe4e852bce64419841875b38afeee91ce WatchSource:0}: Error finding container 48d795b1e5f0513050342c747b329fabe4e852bce64419841875b38afeee91ce: Status 404 returned error can't find the container with id 48d795b1e5f0513050342c747b329fabe4e852bce64419841875b38afeee91ce
Nov 03 07:27:15.385084 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:15.385043    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt" event={"ID":"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee","Type":"ContainerStarted","Data":"88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217"}
Nov 03 07:27:15.385497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:15.385168    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt" event={"ID":"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee","Type":"ContainerStarted","Data":"48d795b1e5f0513050342c747b329fabe4e852bce64419841875b38afeee91ce"}
Nov 03 07:27:15.385497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:15.385313    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:27:15.521386 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:15.520375    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt" podStartSLOduration=2.520354705 podStartE2EDuration="2.520354705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:27:15.517441201 +0000 UTC m=+1687.849810210" watchObservedRunningTime="2026-11-03 07:27:15.520354705 +0000 UTC m=+1687.852723719"
Nov 03 07:27:15.811842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:15.811760    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a0ace7-837c-46b9-9412-0b62601b14c2" path="/var/lib/kubelet/pods/c5a0ace7-837c-46b9-9412-0b62601b14c2/volumes"
Nov 03 07:27:16.060587 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:16.060554    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"]
Nov 03 07:27:16.110806 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:16.110769    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8613f5c-fkh6r"]
Nov 03 07:27:17.810907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:17.810872    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830f0ede-e9b2-4b1f-ad01-9e9371230691" path="/var/lib/kubelet/pods/830f0ede-e9b2-4b1f-ad01-9e9371230691/volumes"
Nov 03 07:27:18.063408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:18.063380    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"]
Nov 03 07:27:18.113569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:18.113530    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ab93a6-287w5"]
Nov 03 07:27:19.808649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:19.808608    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4191ff27-a58d-406a-948b-638274197255" path="/var/lib/kubelet/pods/4191ff27-a58d-406a-948b-638274197255/volumes"
Nov 03 07:27:26.397026 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:26.396989    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:27:27.357467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.357435    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"]
Nov 03 07:27:27.365304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.365269    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:27.422464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.422430    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"]
Nov 03 07:27:27.467638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.467599    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:27:27.467837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.467636    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:27:27.467837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.467780    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:27:27.471742 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.471725    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:27:27.487249 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.487213    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzx9\" (UniqueName: \"kubernetes.io/projected/4382e304-4c30-49ff-9356-0b5bd751ef92-kube-api-access-gbzx9\") pod \"maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:27.487486 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.487465    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:27.487590 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.487570    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:27.589628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.589597    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:27.589807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.589682    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:27.589807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.589786    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzx9\" (UniqueName: \"kubernetes.io/projected/4382e304-4c30-49ff-9356-0b5bd751ef92-kube-api-access-gbzx9\") pod \"maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:27.590276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.590246    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:27.592219 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.592201    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:27.654964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.654894    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzx9\" (UniqueName: \"kubernetes.io/projected/4382e304-4c30-49ff-9356-0b5bd751ef92-kube-api-access-gbzx9\") pod \"maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:27.684987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:27.684948    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:28.070103 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:27:28.070071    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4382e304_4c30_49ff_9356_0b5bd751ef92.slice/crio-eecabdc68d8ca2775bd625ec7b2a7dce462c248a23100521d015b5c7054d51bb WatchSource:0}: Error finding container eecabdc68d8ca2775bd625ec7b2a7dce462c248a23100521d015b5c7054d51bb: Status 404 returned error can't find the container with id eecabdc68d8ca2775bd625ec7b2a7dce462c248a23100521d015b5c7054d51bb
Nov 03 07:27:28.070403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:28.070383    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"]
Nov 03 07:27:28.499094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:28.499059    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8" event={"ID":"4382e304-4c30-49ff-9356-0b5bd751ef92","Type":"ContainerStarted","Data":"01d7226703108514cb6045f9c2a797e1efbd7640c80b0684e133d8080e2a8d46"}
Nov 03 07:27:28.499540 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:28.499110    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8" event={"ID":"4382e304-4c30-49ff-9356-0b5bd751ef92","Type":"ContainerStarted","Data":"eecabdc68d8ca2775bd625ec7b2a7dce462c248a23100521d015b5c7054d51bb"}
Nov 03 07:27:28.623417 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:28.623363    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8" podStartSLOduration=1.6233485650000001 podStartE2EDuration="1.623348565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:27:28.619548481 +0000 UTC m=+1700.951917473" watchObservedRunningTime="2026-11-03 07:27:28.623348565 +0000 UTC m=+1700.955717575"
Nov 03 07:27:29.059072 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:29.059046    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"]
Nov 03 07:27:29.063517 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:29.063492    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18231cb1-jvzgs"]
Nov 03 07:27:29.508926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:29.508896    4470 generic.go:356] "Generic (PLEG): container finished" podID="4382e304-4c30-49ff-9356-0b5bd751ef92" containerID="01d7226703108514cb6045f9c2a797e1efbd7640c80b0684e133d8080e2a8d46" exitCode=0
Nov 03 07:27:29.509394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:29.508985    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8" event={"ID":"4382e304-4c30-49ff-9356-0b5bd751ef92","Type":"ContainerDied","Data":"01d7226703108514cb6045f9c2a797e1efbd7640c80b0684e133d8080e2a8d46"}
Nov 03 07:27:29.808875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:29.808801    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3003102d-67e5-480d-8f93-401e01e2f017" path="/var/lib/kubelet/pods/3003102d-67e5-480d-8f93-401e01e2f017/volumes"
Nov 03 07:27:30.060307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.060231    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"]
Nov 03 07:27:30.110732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.110696    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a07b01f-cd26q"]
Nov 03 07:27:30.687270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.687246    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:30.842450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.842415    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-xks-ca\") pod \"4382e304-4c30-49ff-9356-0b5bd751ef92\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") "
Nov 03 07:27:30.842649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.842555    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-openshift-ca\") pod \"4382e304-4c30-49ff-9356-0b5bd751ef92\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") "
Nov 03 07:27:30.842824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.842798    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4382e304-4c30-49ff-9356-0b5bd751ef92-kube-api-access-gbzx9\" (UniqueName: \"kubernetes.io/projected/4382e304-4c30-49ff-9356-0b5bd751ef92-kube-api-access-gbzx9\") pod \"4382e304-4c30-49ff-9356-0b5bd751ef92\" (UID: \"4382e304-4c30-49ff-9356-0b5bd751ef92\") "
Nov 03 07:27:30.843236 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.843209    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-openshift-ca" pod "4382e304-4c30-49ff-9356-0b5bd751ef92" (UID: "4382e304-4c30-49ff-9356-0b5bd751ef92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:27:30.843654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.843631    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:27:30.845732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.845667    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-xks-ca" pod "4382e304-4c30-49ff-9356-0b5bd751ef92" (UID: "4382e304-4c30-49ff-9356-0b5bd751ef92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:27:30.846044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.846001    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4382e304-4c30-49ff-9356-0b5bd751ef92-kube-api-access-gbzx9" pod "4382e304-4c30-49ff-9356-0b5bd751ef92" (UID: "4382e304-4c30-49ff-9356-0b5bd751ef92"). InnerVolumeSpecName "kube-api-access-gbzx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:27:30.945264 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.945230    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzx9\" (UniqueName: \"kubernetes.io/projected/4382e304-4c30-49ff-9356-0b5bd751ef92-kube-api-access-gbzx9\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:27:30.945264 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:30.945258    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4382e304-4c30-49ff-9356-0b5bd751ef92-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:27:31.526000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:31.525973    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"
Nov 03 07:27:31.526198 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:31.526019    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8" event={"ID":"4382e304-4c30-49ff-9356-0b5bd751ef92","Type":"ContainerDied","Data":"eecabdc68d8ca2775bd625ec7b2a7dce462c248a23100521d015b5c7054d51bb"}
Nov 03 07:27:31.526198 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:31.526102    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eecabdc68d8ca2775bd625ec7b2a7dce462c248a23100521d015b5c7054d51bb"
Nov 03 07:27:31.809216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:31.809121    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad" path="/var/lib/kubelet/pods/61d7bdf5-13f4-4dc4-ba47-e6da58cd3bad/volumes"
Nov 03 07:27:33.134775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:33.134741    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"]
Nov 03 07:27:33.149294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:33.149261    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9f39a0-v5kml"]
Nov 03 07:27:33.809111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:33.809072    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae19d29c-66cf-4d85-8500-8880f6aad534" path="/var/lib/kubelet/pods/ae19d29c-66cf-4d85-8500-8880f6aad534/volumes"
Nov 03 07:27:34.069177 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:34.069116    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"]
Nov 03 07:27:34.120297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:34.120262    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51be60f0-vrqvh"]
Nov 03 07:27:34.760501 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:27:34.760466    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:27:35.808468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:35.808439    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88f03e02-0ff9-443c-91c2-845857d60d8b" path="/var/lib/kubelet/pods/88f03e02-0ff9-443c-91c2-845857d60d8b/volumes"
Nov 03 07:27:37.059126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:37.059091    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"]
Nov 03 07:27:37.109810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:37.109775    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6a6d68-f6fcr"]
Nov 03 07:27:37.810057 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:37.810022    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8" path="/var/lib/kubelet/pods/00c897f4-a4ac-4d35-8ce7-1a7d82ff4fe8/volumes"
Nov 03 07:27:38.058532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.058500    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"]
Nov 03 07:27:38.062564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.062545    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ab3f37-g44w9"]
Nov 03 07:27:38.443514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.443432    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"]
Nov 03 07:27:38.444203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.444188    4470 cpu_manager.go:414] "containerMap: removing container" podUID="4382e304-4c30-49ff-9356-0b5bd751ef92" containerName="revoke-keys"
Nov 03 07:27:38.444251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.444204    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4382e304-4c30-49ff-9356-0b5bd751ef92" containerName="revoke-keys"
Nov 03 07:27:38.444353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.444344    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="4382e304-4c30-49ff-9356-0b5bd751ef92" containerName="revoke-keys"
Nov 03 07:27:38.449443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.449427    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:38.507029 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.506982    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"]
Nov 03 07:27:38.551352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.551309    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:38.551562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.551448    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:38.551681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.551656    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8jd\" (UniqueName: \"kubernetes.io/projected/282b2a7b-8256-4f31-9fca-df1eb45766c0-kube-api-access-kn8jd\") pod \"maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:38.552214 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.552176    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:27:38.552332 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.552243    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:27:38.552332 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.552280    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:27:38.552332 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.552287    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:27:38.653533 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.653502    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:38.653849 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.653627    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8jd\" (UniqueName: \"kubernetes.io/projected/282b2a7b-8256-4f31-9fca-df1eb45766c0-kube-api-access-kn8jd\") pod \"maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:38.653849 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.653803    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:38.654339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.654313    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:38.656336 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.656316    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:38.718331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.718255    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8jd\" (UniqueName: \"kubernetes.io/projected/282b2a7b-8256-4f31-9fca-df1eb45766c0-kube-api-access-kn8jd\") pod \"maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:38.764861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:38.764830    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:39.154180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:39.154150    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"]
Nov 03 07:27:39.597753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:39.597719    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt" event={"ID":"282b2a7b-8256-4f31-9fca-df1eb45766c0","Type":"ContainerStarted","Data":"a013451cf5a4b03bc0b3f14fca6395f365998d40e3c2c5007c0950e073a5cb62"}
Nov 03 07:27:39.808830 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:39.808754    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4534034-0f85-4eaf-8fc6-11c8c130c739" path="/var/lib/kubelet/pods/c4534034-0f85-4eaf-8fc6-11c8c130c739/volumes"
Nov 03 07:27:40.608701 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:40.608670    4470 generic.go:356] "Generic (PLEG): container finished" podID="282b2a7b-8256-4f31-9fca-df1eb45766c0" containerID="e4b8f84676844a9259acb6aab1cf15b68f71a4e9bdcc327ffa9eca82b504d965" exitCode=0
Nov 03 07:27:40.609111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:40.608754    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt" event={"ID":"282b2a7b-8256-4f31-9fca-df1eb45766c0","Type":"ContainerDied","Data":"e4b8f84676844a9259acb6aab1cf15b68f71a4e9bdcc327ffa9eca82b504d965"}
Nov 03 07:27:41.783071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:41.783045    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:41.818269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:41.818236    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282b2a7b-8256-4f31-9fca-df1eb45766c0-kube-api-access-kn8jd\" (UniqueName: \"kubernetes.io/projected/282b2a7b-8256-4f31-9fca-df1eb45766c0-kube-api-access-kn8jd\") pod \"282b2a7b-8256-4f31-9fca-df1eb45766c0\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") "
Nov 03 07:27:41.818428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:41.818351    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-xks-ca\") pod \"282b2a7b-8256-4f31-9fca-df1eb45766c0\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") "
Nov 03 07:27:41.818624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:41.818603    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-openshift-ca\") pod \"282b2a7b-8256-4f31-9fca-df1eb45766c0\" (UID: \"282b2a7b-8256-4f31-9fca-df1eb45766c0\") "
Nov 03 07:27:41.819323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:41.819010    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-openshift-ca" pod "282b2a7b-8256-4f31-9fca-df1eb45766c0" (UID: "282b2a7b-8256-4f31-9fca-df1eb45766c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:27:41.819323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:41.819296    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:27:41.820699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:41.820678    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-xks-ca" pod "282b2a7b-8256-4f31-9fca-df1eb45766c0" (UID: "282b2a7b-8256-4f31-9fca-df1eb45766c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:27:41.820821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:41.820745    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282b2a7b-8256-4f31-9fca-df1eb45766c0-kube-api-access-kn8jd" pod "282b2a7b-8256-4f31-9fca-df1eb45766c0" (UID: "282b2a7b-8256-4f31-9fca-df1eb45766c0"). InnerVolumeSpecName "kube-api-access-kn8jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:27:41.921312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:41.921089    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn8jd\" (UniqueName: \"kubernetes.io/projected/282b2a7b-8256-4f31-9fca-df1eb45766c0-kube-api-access-kn8jd\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:27:41.921524 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:41.921504    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282b2a7b-8256-4f31-9fca-df1eb45766c0-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:27:42.627716 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:42.627688    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"
Nov 03 07:27:42.627895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:42.627733    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt" event={"ID":"282b2a7b-8256-4f31-9fca-df1eb45766c0","Type":"ContainerDied","Data":"a013451cf5a4b03bc0b3f14fca6395f365998d40e3c2c5007c0950e073a5cb62"}
Nov 03 07:27:42.627895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:42.627803    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a013451cf5a4b03bc0b3f14fca6395f365998d40e3c2c5007c0950e073a5cb62"
Nov 03 07:27:43.059126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:43.059042    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"]
Nov 03 07:27:43.063910 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:43.063876    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9f69e6-9g4wd"]
Nov 03 07:27:43.809026 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:43.808990    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea16936-8a96-44c1-a776-27c9c7cd6d70" path="/var/lib/kubelet/pods/8ea16936-8a96-44c1-a776-27c9c7cd6d70/volumes"
Nov 03 07:27:45.061728 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:45.061658    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"]
Nov 03 07:27:45.111790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:45.111757    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce89b84-tmpdt"]
Nov 03 07:27:45.808227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:45.808195    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e9755c-f3e3-4eae-ab90-fd646155683f" path="/var/lib/kubelet/pods/e7e9755c-f3e3-4eae-ab90-fd646155683f/volumes"
Nov 03 07:27:48.059700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:48.059658    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"]
Nov 03 07:27:48.110227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:48.110197    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318aea9-kdsfz"]
Nov 03 07:27:49.809435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:49.809403    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf70f482-a8dc-4d16-84cb-07adad43e3e5" path="/var/lib/kubelet/pods/bf70f482-a8dc-4d16-84cb-07adad43e3e5/volumes"
Nov 03 07:27:54.063339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:54.063303    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"]
Nov 03 07:27:54.112188 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:54.112152    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99cf8fb-5rdhr"]
Nov 03 07:27:55.807710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:27:55.807673    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b" path="/var/lib/kubelet/pods/c15758a2-9ff6-48d3-a8c7-8ecf60d41e3b/volumes"
Nov 03 07:28:04.761342 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:28:04.761306    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:28:11.198918 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:11.198879    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"]
Nov 03 07:28:11.199474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:11.199101    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy" containerID="cri-o://da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2" gracePeriod=30
Nov 03 07:28:11.249243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:11.249207    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"]
Nov 03 07:28:11.249449 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:11.249428    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy" containerID="cri-o://28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b" gracePeriod=30
Nov 03 07:28:13.030988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.030954    4470 scope.go:122] "RemoveContainer" containerID="ceecac0b89a40c9dc07502fdb5241aaec133a8b0cdb50dfe5bcd17aaaee492f9"
Nov 03 07:28:13.047562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.047410    4470 scope.go:122] "RemoveContainer" containerID="23aaf63144e9883dae2132c8201f0d92544513c8de53b357310080e4f069d42a"
Nov 03 07:28:13.063893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.063827    4470 scope.go:122] "RemoveContainer" containerID="f3efac7ccb1ff4402e37f0c4180ab0534d6f830cbfa6b76ae9fff07be1805e47"
Nov 03 07:28:13.081299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.081274    4470 scope.go:122] "RemoveContainer" containerID="b9462f35c428dfc922fe30d9205ca30c88e02b2c2f6dad4164b34f789c121b4e"
Nov 03 07:28:13.099858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.099822    4470 scope.go:122] "RemoveContainer" containerID="cf46938ff0a2623485b1d85d4a9f62aff051d268f6492f476ab9027ddc1f4036"
Nov 03 07:28:13.114581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.114555    4470 scope.go:122] "RemoveContainer" containerID="10b8afa2804108df1ca5a4c59f93ae291ea29c55b8e9a9d724a2f430ee05c44f"
Nov 03 07:28:13.126089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.126067    4470 scope.go:122] "RemoveContainer" containerID="fa6d280236f8102ec78dc7cd43a688cb93e8890711d298993d43fdad1375a98f"
Nov 03 07:28:13.136839 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.136813    4470 scope.go:122] "RemoveContainer" containerID="68ce5c7fb61cb1f56ee652d05b77338ab0635ede55e1ca66010c25297d33455f"
Nov 03 07:28:13.149763 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.149740    4470 scope.go:122] "RemoveContainer" containerID="ae2921e0998c4263bd328de4731a6a74e2fcaefc43abc80e9aa0b1ad12408245"
Nov 03 07:28:13.162120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.162096    4470 scope.go:122] "RemoveContainer" containerID="3892f9452b98a85e88bdf55fb4475a43a0c4760bd4d5c94b7cb7fa5c379e0d96"
Nov 03 07:28:13.176012 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.175989    4470 scope.go:122] "RemoveContainer" containerID="eed6524d7546f55eee74c85270dcc15e5bcebaf86667a04e9d051fc5ce655ea3"
Nov 03 07:28:13.189305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.189282    4470 scope.go:122] "RemoveContainer" containerID="953c7ed42a9663ca4ec7d6ff1271c74bf7a96993eb967d6c82f113575e5048f8"
Nov 03 07:28:13.200313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.200289    4470 scope.go:122] "RemoveContainer" containerID="2d72e097bf4c93fbeb8762a66eece611456176c12f0c71cc40b87b850050c4e0"
Nov 03 07:28:13.216414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:13.216387    4470 scope.go:122] "RemoveContainer" containerID="bf784798dbed856eb9ce9ebf032f6559cc29fbc02ca7d429647e6d2f174ba639"
Nov 03 07:28:15.060063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:15.060026    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"]
Nov 03 07:28:15.108528 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:15.108495    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-556526bc-jhbjr"]
Nov 03 07:28:15.808781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:15.808745    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6538b2-8e39-461f-b0aa-95943a811c34" path="/var/lib/kubelet/pods/5d6538b2-8e39-461f-b0aa-95943a811c34/volumes"
Nov 03 07:28:16.567373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.567348    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:28:16.606230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.606208    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:28:16.651344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651306    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-socket\") pod \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") "
Nov 03 07:28:16.651535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651395    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-data\") pod \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") "
Nov 03 07:28:16.651535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651421    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-certs\") pod \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") "
Nov 03 07:28:16.651665 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651612    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-kube-api-access-678s7\" (UniqueName: \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-kube-api-access-678s7\") pod \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") "
Nov 03 07:28:16.651665 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651619    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-data" pod "63c21c33-c6b9-44ee-9b22-0b1951c3d888" (UID: "63c21c33-c6b9-44ee-9b22-0b1951c3d888"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:16.651665 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651635    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-socket" pod "63c21c33-c6b9-44ee-9b22-0b1951c3d888" (UID: "63c21c33-c6b9-44ee-9b22-0b1951c3d888"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:16.651665 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651662    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-token\" (UniqueName: \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-token\") pod \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") "
Nov 03 07:28:16.651873 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651730    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-certs" pod "63c21c33-c6b9-44ee-9b22-0b1951c3d888" (UID: "63c21c33-c6b9-44ee-9b22-0b1951c3d888"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:16.651873 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651741    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-envoy\") pod \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") "
Nov 03 07:28:16.651873 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651785    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istiod-ca-cert\") pod \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") "
Nov 03 07:28:16.651873 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651831    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-credential-socket\") pod \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") "
Nov 03 07:28:16.652064 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.651880    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-podinfo\") pod \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\" (UID: \"63c21c33-c6b9-44ee-9b22-0b1951c3d888\") "
Nov 03 07:28:16.652169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.652101    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-credential-socket" pod "63c21c33-c6b9-44ee-9b22-0b1951c3d888" (UID: "63c21c33-c6b9-44ee-9b22-0b1951c3d888"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:16.652169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.652106    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istiod-ca-cert" pod "63c21c33-c6b9-44ee-9b22-0b1951c3d888" (UID: "63c21c33-c6b9-44ee-9b22-0b1951c3d888"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:28:16.652565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.652544    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.652565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.652571    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.652709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.652585    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.652709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.652597    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.652709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.652611    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.654013 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.653995    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-token" pod "63c21c33-c6b9-44ee-9b22-0b1951c3d888" (UID: "63c21c33-c6b9-44ee-9b22-0b1951c3d888"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:28:16.654013 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.653997    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-envoy" pod "63c21c33-c6b9-44ee-9b22-0b1951c3d888" (UID: "63c21c33-c6b9-44ee-9b22-0b1951c3d888"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:16.654562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.654545    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-podinfo" pod "63c21c33-c6b9-44ee-9b22-0b1951c3d888" (UID: "63c21c33-c6b9-44ee-9b22-0b1951c3d888"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:28:16.654631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.654565    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-kube-api-access-678s7" pod "63c21c33-c6b9-44ee-9b22-0b1951c3d888" (UID: "63c21c33-c6b9-44ee-9b22-0b1951c3d888"). InnerVolumeSpecName "kube-api-access-678s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:28:16.753818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.753781    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-data\") pod \"7ba92ef7-803c-4082-8641-c4d65e64b615\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") "
Nov 03 07:28:16.754007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.753835    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-token\" (UniqueName: \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-token\") pod \"7ba92ef7-803c-4082-8641-c4d65e64b615\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") "
Nov 03 07:28:16.754007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.753883    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba92ef7-803c-4082-8641-c4d65e64b615-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ba92ef7-803c-4082-8641-c4d65e64b615-istiod-ca-cert\") pod \"7ba92ef7-803c-4082-8641-c4d65e64b615\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") "
Nov 03 07:28:16.754007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.753966    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-envoy\") pod \"7ba92ef7-803c-4082-8641-c4d65e64b615\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") "
Nov 03 07:28:16.754224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.754035    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-socket\") pod \"7ba92ef7-803c-4082-8641-c4d65e64b615\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") "
Nov 03 07:28:16.754224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.754041    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-data" pod "7ba92ef7-803c-4082-8641-c4d65e64b615" (UID: "7ba92ef7-803c-4082-8641-c4d65e64b615"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:16.754224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.754154    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-credential-socket\") pod \"7ba92ef7-803c-4082-8641-c4d65e64b615\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") "
Nov 03 07:28:16.754224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.754209    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-kube-api-access-qs6d5\" (UniqueName: \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-kube-api-access-qs6d5\") pod \"7ba92ef7-803c-4082-8641-c4d65e64b615\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") "
Nov 03 07:28:16.754458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.754277    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-podinfo\") pod \"7ba92ef7-803c-4082-8641-c4d65e64b615\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") "
Nov 03 07:28:16.754458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.754402    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-socket" pod "7ba92ef7-803c-4082-8641-c4d65e64b615" (UID: "7ba92ef7-803c-4082-8641-c4d65e64b615"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:16.754458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.754447    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-certs\") pod \"7ba92ef7-803c-4082-8641-c4d65e64b615\" (UID: \"7ba92ef7-803c-4082-8641-c4d65e64b615\") "
Nov 03 07:28:16.754634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.754475    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba92ef7-803c-4082-8641-c4d65e64b615-istiod-ca-cert" pod "7ba92ef7-803c-4082-8641-c4d65e64b615" (UID: "7ba92ef7-803c-4082-8641-c4d65e64b615"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:28:16.754634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.754498    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-credential-socket" pod "7ba92ef7-803c-4082-8641-c4d65e64b615" (UID: "7ba92ef7-803c-4082-8641-c4d65e64b615"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:16.754795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.754768    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-certs" pod "7ba92ef7-803c-4082-8641-c4d65e64b615" (UID: "7ba92ef7-803c-4082-8641-c4d65e64b615"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:16.755406 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.755380    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.755406 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.755403    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.755580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.755417    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.755580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.755430    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.755580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.755456    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ba92ef7-803c-4082-8641-c4d65e64b615-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.755580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.755470    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-678s7\" (UniqueName: \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-kube-api-access-678s7\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.755580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.755483    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.755580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.755497    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.755580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.755510    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c21c33-c6b9-44ee-9b22-0b1951c3d888-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.756957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.756902    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-envoy" pod "7ba92ef7-803c-4082-8641-c4d65e64b615" (UID: "7ba92ef7-803c-4082-8641-c4d65e64b615"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:16.757218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.757201    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-kube-api-access-qs6d5" pod "7ba92ef7-803c-4082-8641-c4d65e64b615" (UID: "7ba92ef7-803c-4082-8641-c4d65e64b615"). InnerVolumeSpecName "kube-api-access-qs6d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:28:16.757619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.757596    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-podinfo" pod "7ba92ef7-803c-4082-8641-c4d65e64b615" (UID: "7ba92ef7-803c-4082-8641-c4d65e64b615"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:28:16.758294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.758268    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-token" pod "7ba92ef7-803c-4082-8641-c4d65e64b615" (UID: "7ba92ef7-803c-4082-8641-c4d65e64b615"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:28:16.862987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.862946    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.862987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.862987    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.863283 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.863000    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ba92ef7-803c-4082-8641-c4d65e64b615-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.863283 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.863013    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs6d5\" (UniqueName: \"kubernetes.io/projected/7ba92ef7-803c-4082-8641-c4d65e64b615-kube-api-access-qs6d5\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:16.973761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.973726    4470 generic.go:356] "Generic (PLEG): container finished" podID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerID="28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b" exitCode=0
Nov 03 07:28:16.973960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.973775    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"
Nov 03 07:28:16.973960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.973811    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" event={"ID":"7ba92ef7-803c-4082-8641-c4d65e64b615","Type":"ContainerDied","Data":"28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b"}
Nov 03 07:28:16.973960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.973856    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj" event={"ID":"7ba92ef7-803c-4082-8641-c4d65e64b615","Type":"ContainerDied","Data":"6f0c8b065d15dc4f823eef0cc7b84dfe289641e3bbfcec5c562808aa6e8e5d12"}
Nov 03 07:28:16.973960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.973870    4470 scope.go:122] "RemoveContainer" containerID="28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b"
Nov 03 07:28:16.976289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.976267    4470 generic.go:356] "Generic (PLEG): container finished" podID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerID="da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2" exitCode=0
Nov 03 07:28:16.976406 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.976325    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"
Nov 03 07:28:16.976406 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.976348    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" event={"ID":"63c21c33-c6b9-44ee-9b22-0b1951c3d888","Type":"ContainerDied","Data":"da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2"}
Nov 03 07:28:16.976490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:16.976416    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j" event={"ID":"63c21c33-c6b9-44ee-9b22-0b1951c3d888","Type":"ContainerDied","Data":"a41c0819907ae55c0c1528065e1e93181798c5e3e11ba8bade0b422aaeb5ff63"}
Nov 03 07:28:17.003051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.001539    4470 scope.go:122] "RemoveContainer" containerID="28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b"
Nov 03 07:28:17.003051 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:28:17.001955    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b\": container with ID starting with 28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b not found: ID does not exist" containerID="28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b"
Nov 03 07:28:17.003051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.002000    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b"} err="failed to get container status \"28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b\": rpc error: code = NotFound desc = could not find container \"28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b\": container with ID starting with 28154f6788d255e88f252c47b82ed3474a40aaad7b425feedb66da662d4ed36b not found: ID does not exist"
Nov 03 07:28:17.003051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.002023    4470 scope.go:122] "RemoveContainer" containerID="da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2"
Nov 03 07:28:17.016312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.016291    4470 scope.go:122] "RemoveContainer" containerID="c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34"
Nov 03 07:28:17.026676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.026653    4470 scope.go:122] "RemoveContainer" containerID="da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2"
Nov 03 07:28:17.026971 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:28:17.026940    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2\": container with ID starting with da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2 not found: ID does not exist" containerID="da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2"
Nov 03 07:28:17.027063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.026974    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2"} err="failed to get container status \"da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2\": rpc error: code = NotFound desc = could not find container \"da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2\": container with ID starting with da16d803a54dba6f8ce9f9a7e67c5c1c9f916333f09c2d55ade25ed56979bcb2 not found: ID does not exist"
Nov 03 07:28:17.027063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.027002    4470 scope.go:122] "RemoveContainer" containerID="c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34"
Nov 03 07:28:17.027270 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:28:17.027252    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34\": container with ID starting with c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34 not found: ID does not exist" containerID="c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34"
Nov 03 07:28:17.027336 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.027281    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34"} err="failed to get container status \"c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34\": rpc error: code = NotFound desc = could not find container \"c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34\": container with ID starting with c2431b82fa613c5605c617464096b96ff443180a866ab41733e84527e24fae34 not found: ID does not exist"
Nov 03 07:28:17.151000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.150963    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"]
Nov 03 07:28:17.155052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.155026    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj"]
Nov 03 07:28:17.273078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.272997    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"]
Nov 03 07:28:17.277065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.277045    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j"]
Nov 03 07:28:17.812550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.812514    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" path="/var/lib/kubelet/pods/63c21c33-c6b9-44ee-9b22-0b1951c3d888/volumes"
Nov 03 07:28:17.813247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:17.813229    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" path="/var/lib/kubelet/pods/7ba92ef7-803c-4082-8641-c4d65e64b615/volumes"
Nov 03 07:28:26.524708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:26.524671    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"]
Nov 03 07:28:26.525180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:26.524897    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" podUID="5255b255-a1c9-4da0-b31f-016adf8c066b" containerName="istio-proxy" containerID="cri-o://98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522" gracePeriod=30
Nov 03 07:28:26.577829 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:26.577779    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"]
Nov 03 07:28:26.578042 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:26.578016    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" containerID="cri-o://399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770" gracePeriod=30
Nov 03 07:28:26.830585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:26.830551    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"]
Nov 03 07:28:26.834382 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:26.834361    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d089af5-h2hv8"]
Nov 03 07:28:27.815603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:27.815564    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4382e304-4c30-49ff-9356-0b5bd751ef92" path="/var/lib/kubelet/pods/4382e304-4c30-49ff-9356-0b5bd751ef92/volumes"
Nov 03 07:28:28.254235 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.254126    4470 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:28:28.254235 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.254212    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:28:28.968228 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.968194    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"]
Nov 03 07:28:28.981005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.980964    4470 cpu_manager.go:414] "containerMap: removing container" podUID="282b2a7b-8256-4f31-9fca-df1eb45766c0" containerName="revoke-keys"
Nov 03 07:28:28.981005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.980996    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282b2a7b-8256-4f31-9fca-df1eb45766c0" containerName="revoke-keys"
Nov 03 07:28:28.981250 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.981048    4470 cpu_manager.go:414] "containerMap: removing container" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy"
Nov 03 07:28:28.981250 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.981054    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy"
Nov 03 07:28:28.981250 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.981064    4470 cpu_manager.go:414] "containerMap: removing container" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy"
Nov 03 07:28:28.981250 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.981069    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy"
Nov 03 07:28:28.981250 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.981076    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy"
Nov 03 07:28:28.981250 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.981080    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy"
Nov 03 07:28:28.981659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.981376    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy"
Nov 03 07:28:28.981659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.981396    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c21c33-c6b9-44ee-9b22-0b1951c3d888" containerName="istio-proxy"
Nov 03 07:28:28.981659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.981408    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba92ef7-803c-4082-8641-c4d65e64b615" containerName="istio-proxy"
Nov 03 07:28:28.981659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:28.981423    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="282b2a7b-8256-4f31-9fca-df1eb45766c0" containerName="revoke-keys"
Nov 03 07:28:29.011458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.011426    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.086831 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.086577    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-5183ae-openshift-default-dockercfg-vzx5s\""
Nov 03 07:28:29.245573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.229321    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"]
Nov 03 07:28:29.246613 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.246576    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-envoy\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.246830 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.246815    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-token\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.247030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.247015    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-podinfo\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.247312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.247288    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-certs\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.247553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.247536    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-socket\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.247724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.247707    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabd31f7-12a2-4753-8a37-bc835e84615d-istiod-ca-cert\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.247844 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.247830    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-data\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.247959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.247945    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-credential-socket\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.248062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.248047    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nhdg\" (UniqueName: \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-kube-api-access-9nhdg\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.349950 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.349902    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-envoy\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.350153 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.349979    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-token\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.350153 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.350071    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-podinfo\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.350303 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.350178    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-certs\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.350303 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.350258    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-socket\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.350408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.350308    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabd31f7-12a2-4753-8a37-bc835e84615d-istiod-ca-cert\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.350408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.350334    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-data\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.350408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.350371    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-credential-socket\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.350575 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.350407    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nhdg\" (UniqueName: \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-kube-api-access-9nhdg\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.350676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.350650    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-certs\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.351313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.351288    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-socket\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.351445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.351313    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-credential-socket\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.351445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.351354    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabd31f7-12a2-4753-8a37-bc835e84615d-istiod-ca-cert\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.351561 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.351528    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-data\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.353015 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.352989    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-envoy\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.353273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.353256    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-podinfo\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.416816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.416700    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-token\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.417621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.417594    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nhdg\" (UniqueName: \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-kube-api-access-9nhdg\") pod \"e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") " pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.542307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.542223    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:29.992774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.992745    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"]
Nov 03 07:28:29.995404 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:28:29.995373    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabd31f7_12a2_4753_8a37_bc835e84615d.slice/crio-45ed25babd4bde5c17bc6e2e3c97a3b74a6bcc5b8a7edf45c9c38f5a630dc9ac WatchSource:0}: Error finding container 45ed25babd4bde5c17bc6e2e3c97a3b74a6bcc5b8a7edf45c9c38f5a630dc9ac: Status 404 returned error can't find the container with id 45ed25babd4bde5c17bc6e2e3c97a3b74a6bcc5b8a7edf45c9c38f5a630dc9ac
Nov 03 07:28:29.998519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.998475    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:28:29.998628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.998576    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:28:29.998680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:29.998626    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:28:30.283417 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:30.283325    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" event={"ID":"eabd31f7-12a2-4753-8a37-bc835e84615d","Type":"ContainerStarted","Data":"35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d"}
Nov 03 07:28:30.283417 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:30.283380    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" event={"ID":"eabd31f7-12a2-4753-8a37-bc835e84615d","Type":"ContainerStarted","Data":"45ed25babd4bde5c17bc6e2e3c97a3b74a6bcc5b8a7edf45c9c38f5a630dc9ac"}
Nov 03 07:28:30.411969 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:30.411914    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" podStartSLOduration=2.4118995 podStartE2EDuration="2.4118995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:30.408561363 +0000 UTC m=+1762.740930358" watchObservedRunningTime="2026-11-03 07:28:30.4118995 +0000 UTC m=+1762.744268493"
Nov 03 07:28:30.546448 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:30.546360    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:31.549189 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:31.549115    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:31.931190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:31.930810    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:28:32.004714 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:31.999476    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:28:32.010561 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.010536    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-data\") pod \"5255b255-a1c9-4da0-b31f-016adf8c066b\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") "
Nov 03 07:28:32.010718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.010626    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-token\" (UniqueName: \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-token\") pod \"5255b255-a1c9-4da0-b31f-016adf8c066b\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") "
Nov 03 07:28:32.010778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.010731    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-socket\") pod \"5255b255-a1c9-4da0-b31f-016adf8c066b\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") "
Nov 03 07:28:32.010778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.010759    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-certs\") pod \"5255b255-a1c9-4da0-b31f-016adf8c066b\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") "
Nov 03 07:28:32.010878 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.010810    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-envoy\") pod \"5255b255-a1c9-4da0-b31f-016adf8c066b\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") "
Nov 03 07:28:32.010878 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.010853    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-podinfo\") pod \"5255b255-a1c9-4da0-b31f-016adf8c066b\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") "
Nov 03 07:28:32.010977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.010886    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5255b255-a1c9-4da0-b31f-016adf8c066b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5255b255-a1c9-4da0-b31f-016adf8c066b-istiod-ca-cert\") pod \"5255b255-a1c9-4da0-b31f-016adf8c066b\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") "
Nov 03 07:28:32.010977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.010888    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-data" pod "5255b255-a1c9-4da0-b31f-016adf8c066b" (UID: "5255b255-a1c9-4da0-b31f-016adf8c066b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:32.011078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.011021    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-credential-socket\") pod \"5255b255-a1c9-4da0-b31f-016adf8c066b\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") "
Nov 03 07:28:32.011078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.011030    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-certs" pod "5255b255-a1c9-4da0-b31f-016adf8c066b" (UID: "5255b255-a1c9-4da0-b31f-016adf8c066b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:32.011078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.011056    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-kube-api-access-rkppq\" (UniqueName: \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-kube-api-access-rkppq\") pod \"5255b255-a1c9-4da0-b31f-016adf8c066b\" (UID: \"5255b255-a1c9-4da0-b31f-016adf8c066b\") "
Nov 03 07:28:32.011482 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.011462    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5255b255-a1c9-4da0-b31f-016adf8c066b-istiod-ca-cert" pod "5255b255-a1c9-4da0-b31f-016adf8c066b" (UID: "5255b255-a1c9-4da0-b31f-016adf8c066b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:28:32.011853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.011825    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-credential-socket" pod "5255b255-a1c9-4da0-b31f-016adf8c066b" (UID: "5255b255-a1c9-4da0-b31f-016adf8c066b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:32.011977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.011855    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.011977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.011876    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.011977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.011890    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5255b255-a1c9-4da0-b31f-016adf8c066b-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.012386 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.012361    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-socket" pod "5255b255-a1c9-4da0-b31f-016adf8c066b" (UID: "5255b255-a1c9-4da0-b31f-016adf8c066b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:32.013580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.013558    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-envoy" pod "5255b255-a1c9-4da0-b31f-016adf8c066b" (UID: "5255b255-a1c9-4da0-b31f-016adf8c066b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:32.013671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.013647    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-token" pod "5255b255-a1c9-4da0-b31f-016adf8c066b" (UID: "5255b255-a1c9-4da0-b31f-016adf8c066b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:28:32.013671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.013660    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-kube-api-access-rkppq" pod "5255b255-a1c9-4da0-b31f-016adf8c066b" (UID: "5255b255-a1c9-4da0-b31f-016adf8c066b"). InnerVolumeSpecName "kube-api-access-rkppq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:28:32.014314 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.014287    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-podinfo" pod "5255b255-a1c9-4da0-b31f-016adf8c066b" (UID: "5255b255-a1c9-4da0-b31f-016adf8c066b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:28:32.113521 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.113415    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-envoy\") pod \"43aeb176-2074-480e-8ef5-251d3babd226\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") "
Nov 03 07:28:32.113685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.113577    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-data\") pod \"43aeb176-2074-480e-8ef5-251d3babd226\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") "
Nov 03 07:28:32.113949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.113918    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-socket" pod "43aeb176-2074-480e-8ef5-251d3babd226" (UID: "43aeb176-2074-480e-8ef5-251d3babd226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:32.114276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.114196    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-data" pod "43aeb176-2074-480e-8ef5-251d3babd226" (UID: "43aeb176-2074-480e-8ef5-251d3babd226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:32.114276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.114240    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-socket\") pod \"43aeb176-2074-480e-8ef5-251d3babd226\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") "
Nov 03 07:28:32.114276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.114319    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-credential-socket\") pod \"43aeb176-2074-480e-8ef5-251d3babd226\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") "
Nov 03 07:28:32.114530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.114386    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-istio-token\" (UniqueName: \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-istio-token\") pod \"43aeb176-2074-480e-8ef5-251d3babd226\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") "
Nov 03 07:28:32.114591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.114573    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-credential-socket" pod "43aeb176-2074-480e-8ef5-251d3babd226" (UID: "43aeb176-2074-480e-8ef5-251d3babd226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:32.114656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.114600    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-kube-api-access-zjkkf\" (UniqueName: \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-kube-api-access-zjkkf\") pod \"43aeb176-2074-480e-8ef5-251d3babd226\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") "
Nov 03 07:28:32.114656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.114646    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43aeb176-2074-480e-8ef5-251d3babd226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43aeb176-2074-480e-8ef5-251d3babd226-istiod-ca-cert\") pod \"43aeb176-2074-480e-8ef5-251d3babd226\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") "
Nov 03 07:28:32.114767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.114721    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43aeb176-2074-480e-8ef5-251d3babd226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43aeb176-2074-480e-8ef5-251d3babd226-istio-podinfo\") pod \"43aeb176-2074-480e-8ef5-251d3babd226\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") "
Nov 03 07:28:32.114910 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.114886    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-certs\") pod \"43aeb176-2074-480e-8ef5-251d3babd226\" (UID: \"43aeb176-2074-480e-8ef5-251d3babd226\") "
Nov 03 07:28:32.115003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.114981    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43aeb176-2074-480e-8ef5-251d3babd226-istiod-ca-cert" pod "43aeb176-2074-480e-8ef5-251d3babd226" (UID: "43aeb176-2074-480e-8ef5-251d3babd226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:28:32.115635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.115599    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-certs" pod "43aeb176-2074-480e-8ef5-251d3babd226" (UID: "43aeb176-2074-480e-8ef5-251d3babd226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:32.115826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.115803    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.115826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.115827    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.115999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.115841    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.115999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.115856    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkppq\" (UniqueName: \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-kube-api-access-rkppq\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.115999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.115870    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43aeb176-2074-480e-8ef5-251d3babd226-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.115999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.115884    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.116679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.115897    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.116766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.116692    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.116766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.116714    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5255b255-a1c9-4da0-b31f-016adf8c066b-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.116766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.116730    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.116766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.116755    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.116990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.116845    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-envoy" pod "43aeb176-2074-480e-8ef5-251d3babd226" (UID: "43aeb176-2074-480e-8ef5-251d3babd226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:32.119183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.117655    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-kube-api-access-zjkkf" pod "43aeb176-2074-480e-8ef5-251d3babd226" (UID: "43aeb176-2074-480e-8ef5-251d3babd226"). InnerVolumeSpecName "kube-api-access-zjkkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:28:32.119183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.117676    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43aeb176-2074-480e-8ef5-251d3babd226-istio-podinfo" pod "43aeb176-2074-480e-8ef5-251d3babd226" (UID: "43aeb176-2074-480e-8ef5-251d3babd226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:28:32.119183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.117775    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-istio-token" pod "43aeb176-2074-480e-8ef5-251d3babd226" (UID: "43aeb176-2074-480e-8ef5-251d3babd226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:28:32.246160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.239709    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43aeb176-2074-480e-8ef5-251d3babd226-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.278156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.272513    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.278156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.272584    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjkkf\" (UniqueName: \"kubernetes.io/projected/43aeb176-2074-480e-8ef5-251d3babd226-kube-api-access-zjkkf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.278156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.272603    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43aeb176-2074-480e-8ef5-251d3babd226-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:32.336159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.332106    4470 generic.go:356] "Generic (PLEG): container finished" podID="5255b255-a1c9-4da0-b31f-016adf8c066b" containerID="98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522" exitCode=0
Nov 03 07:28:32.336159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.332216    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" event={"ID":"5255b255-a1c9-4da0-b31f-016adf8c066b","Type":"ContainerDied","Data":"98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522"}
Nov 03 07:28:32.336159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.332288    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr" event={"ID":"5255b255-a1c9-4da0-b31f-016adf8c066b","Type":"ContainerDied","Data":"685bb461f0563851d766e7b0d15a0a2def902fb7130deca5e49e4b5e4a5cd21f"}
Nov 03 07:28:32.336159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.332317    4470 scope.go:122] "RemoveContainer" containerID="98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522"
Nov 03 07:28:32.336159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.332463    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"
Nov 03 07:28:32.360942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.360811    4470 generic.go:356] "Generic (PLEG): container finished" podID="43aeb176-2074-480e-8ef5-251d3babd226" containerID="399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770" exitCode=0
Nov 03 07:28:32.363170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.363124    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"
Nov 03 07:28:32.365999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.365930    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" event={"ID":"43aeb176-2074-480e-8ef5-251d3babd226","Type":"ContainerDied","Data":"399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770"}
Nov 03 07:28:32.366183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.366165    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx" event={"ID":"43aeb176-2074-480e-8ef5-251d3babd226","Type":"ContainerDied","Data":"3ac66ad537a81d4d579f2a0d7521ddfe2c98362dab4c2ea8233e9da5db6bd263"}
Nov 03 07:28:32.366361 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.366345    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:32.367789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.367773    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:28:32.378369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.378325    4470 scope.go:122] "RemoveContainer" containerID="98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522"
Nov 03 07:28:32.393380 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:28:32.393326    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522\": container with ID starting with 98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522 not found: ID does not exist" containerID="98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522"
Nov 03 07:28:32.393573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.393546    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522"} err="failed to get container status \"98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522\": rpc error: code = NotFound desc = could not find container \"98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522\": container with ID starting with 98aad041b10f7c1ee0dc2541e9f32edf92ecfd3f1fdf30aa68bf03a8b9b3c522 not found: ID does not exist"
Nov 03 07:28:32.393661 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.393578    4470 scope.go:122] "RemoveContainer" containerID="399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770"
Nov 03 07:28:32.407297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.407275    4470 scope.go:122] "RemoveContainer" containerID="399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770"
Nov 03 07:28:32.407697 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:28:32.407673    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770\": container with ID starting with 399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770 not found: ID does not exist" containerID="399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770"
Nov 03 07:28:32.407767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.407710    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770"} err="failed to get container status \"399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770\": rpc error: code = NotFound desc = could not find container \"399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770\": container with ID starting with 399b380b9966cc60005347b88b1f591e7a10360c9dea52f9c42cb173f19ba770 not found: ID does not exist"
Nov 03 07:28:32.708565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.708490    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"]
Nov 03 07:28:32.717553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.717530    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx"]
Nov 03 07:28:32.832800 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.832764    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"]
Nov 03 07:28:32.841916 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:32.841888    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr"]
Nov 03 07:28:33.699680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:33.699648    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"]
Nov 03 07:28:33.749677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:33.749641    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-209c4c70-5d9wt"]
Nov 03 07:28:33.809853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:33.809818    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282b2a7b-8256-4f31-9fca-df1eb45766c0" path="/var/lib/kubelet/pods/282b2a7b-8256-4f31-9fca-df1eb45766c0/volumes"
Nov 03 07:28:33.810562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:33.810542    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43aeb176-2074-480e-8ef5-251d3babd226" path="/var/lib/kubelet/pods/43aeb176-2074-480e-8ef5-251d3babd226/volumes"
Nov 03 07:28:33.811176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:33.811160    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5255b255-a1c9-4da0-b31f-016adf8c066b" path="/var/lib/kubelet/pods/5255b255-a1c9-4da0-b31f-016adf8c066b/volumes"
Nov 03 07:28:34.761250 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:28:34.761209    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:28:37.887487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:37.887452    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"]
Nov 03 07:28:37.889828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:37.889797    4470 cpu_manager.go:414] "containerMap: removing container" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy"
Nov 03 07:28:37.891227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:37.891204    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy"
Nov 03 07:28:37.891382 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:37.891370    4470 cpu_manager.go:414] "containerMap: removing container" podUID="5255b255-a1c9-4da0-b31f-016adf8c066b" containerName="istio-proxy"
Nov 03 07:28:37.891468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:37.891457    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5255b255-a1c9-4da0-b31f-016adf8c066b" containerName="istio-proxy"
Nov 03 07:28:37.891809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:37.891790    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="43aeb176-2074-480e-8ef5-251d3babd226" containerName="istio-proxy"
Nov 03 07:28:37.891924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:37.891913    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="5255b255-a1c9-4da0-b31f-016adf8c066b" containerName="istio-proxy"
Nov 03 07:28:37.898595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:37.898574    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:37.986159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:37.978244    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-5bdb10-openshift-default-dockercfg-cbltb\""
Nov 03 07:28:38.017240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.017203    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"]
Nov 03 07:28:38.079158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.054361    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-token\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.079158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.062369    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-credential-socket\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.079557 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.079242    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-socket\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.079557 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.079420    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c5935f-9218-450b-a7d5-8fd56685d917-istiod-ca-cert\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.079557 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.079478    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-podinfo\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.079557 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.079522    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhrtw\" (UniqueName: \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-kube-api-access-jhrtw\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.079856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.079660    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-envoy\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.079856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.079725    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-data\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.079856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.079853    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-certs\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.181490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.181390    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-credential-socket\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.181490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.181456    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-socket\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.181490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.181493    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c5935f-9218-450b-a7d5-8fd56685d917-istiod-ca-cert\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.181791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.181521    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-podinfo\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.181791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.181548    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrtw\" (UniqueName: \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-kube-api-access-jhrtw\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.181791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.181614    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-envoy\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.181791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.181644    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-data\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.181791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.181710    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-certs\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.181791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.181743    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-token\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.182409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.182385    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-credential-socket\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.182936 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.182914    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-socket\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.183120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.183100    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c5935f-9218-450b-a7d5-8fd56685d917-istiod-ca-cert\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.183530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.183507    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-certs\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.183590 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.183544    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-data\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.184814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.184793    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-envoy\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.185813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.185797    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-podinfo\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.230489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.230459    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"]
Nov 03 07:28:38.235985 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.235967    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:28:38.292516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.292483    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-5183ae-dockercfg-tp7v4\""
Nov 03 07:28:38.301158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.293300    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-5183ae\""
Nov 03 07:28:38.301158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.298830    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhrtw\" (UniqueName: \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-kube-api-access-jhrtw\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.301158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.300486    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-token\") pod \"e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") " pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.486281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.486117    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"]
Nov 03 07:28:38.626322 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.606097    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:38.630045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.627635    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2905b11-1697-45de-8261-ff449c0cb075-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr\" (UID: \"e2905b11-1697-45de-8261-ff449c0cb075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:28:38.630045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.627998    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw2g5\" (UniqueName: \"kubernetes.io/projected/e2905b11-1697-45de-8261-ff449c0cb075-kube-api-access-kw2g5\") pod \"payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr\" (UID: \"e2905b11-1697-45de-8261-ff449c0cb075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:28:38.734444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.729820    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2905b11-1697-45de-8261-ff449c0cb075-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr\" (UID: \"e2905b11-1697-45de-8261-ff449c0cb075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:28:38.734444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.730123    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw2g5\" (UniqueName: \"kubernetes.io/projected/e2905b11-1697-45de-8261-ff449c0cb075-kube-api-access-kw2g5\") pod \"payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr\" (UID: \"e2905b11-1697-45de-8261-ff449c0cb075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:28:38.734444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.730873    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2905b11-1697-45de-8261-ff449c0cb075-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr\" (UID: \"e2905b11-1697-45de-8261-ff449c0cb075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:28:38.807163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.801073    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw2g5\" (UniqueName: \"kubernetes.io/projected/e2905b11-1697-45de-8261-ff449c0cb075-kube-api-access-kw2g5\") pod \"payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr\" (UID: \"e2905b11-1697-45de-8261-ff449c0cb075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:28:38.889249 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:38.888862    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:28:39.242713 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:39.242674    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"]
Nov 03 07:28:39.289685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:39.289641    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:28:39.289894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:39.289715    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:28:39.289894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:39.289743    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:28:39.491688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:39.491660    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"]
Nov 03 07:28:39.525744 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:28:39.525687    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2905b11_1697_45de_8261_ff449c0cb075.slice/crio-3a5600ab41fa8acaa464681b0ac45343036d0c2f698f1cbc356177f37d516858 WatchSource:0}: Error finding container 3a5600ab41fa8acaa464681b0ac45343036d0c2f698f1cbc356177f37d516858: Status 404 returned error can't find the container with id 3a5600ab41fa8acaa464681b0ac45343036d0c2f698f1cbc356177f37d516858
Nov 03 07:28:39.718471 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:39.718390    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr" event={"ID":"e2905b11-1697-45de-8261-ff449c0cb075","Type":"ContainerStarted","Data":"3a5600ab41fa8acaa464681b0ac45343036d0c2f698f1cbc356177f37d516858"}
Nov 03 07:28:39.721215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:39.721185    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" event={"ID":"c3c5935f-9218-450b-a7d5-8fd56685d917","Type":"ContainerStarted","Data":"e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba"}
Nov 03 07:28:39.721447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:39.721428    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" event={"ID":"c3c5935f-9218-450b-a7d5-8fd56685d917","Type":"ContainerStarted","Data":"9c997d18a6a76579ff530f7c10df24bfea89c67373fd26f68bd82dd086bb0543"}
Nov 03 07:28:39.943156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:39.933000    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" podStartSLOduration=2.932976444 podStartE2EDuration="2.932976444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:39.892779934 +0000 UTC m=+1772.225148930" watchObservedRunningTime="2026-11-03 07:28:39.932976444 +0000 UTC m=+1772.265345440"
Nov 03 07:28:40.606655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:40.606617    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:40.612389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:40.612366    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:40.731014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:40.730969    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr" event={"ID":"e2905b11-1697-45de-8261-ff449c0cb075","Type":"ContainerStarted","Data":"ea9a769f129d4cb852a739c75493e28780bce4e8eeaea4995dbb09f19fa6bce3"}
Nov 03 07:28:40.731285 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:40.731272    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:28:40.731353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:40.731299    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:40.732404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:40.732384    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:28:40.897701 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:40.897588    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr" podStartSLOduration=2.897574008 podStartE2EDuration="2.897574008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:40.892968079 +0000 UTC m=+1773.225337073" watchObservedRunningTime="2026-11-03 07:28:40.897574008 +0000 UTC m=+1773.229943039"
Nov 03 07:28:41.596251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:41.596218    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"]
Nov 03 07:28:41.596715 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:41.596445    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy" containerID="cri-o://db4ca98833575ec82957daba56192550d1b065c5060119843f7596987232395a" gracePeriod=30
Nov 03 07:28:44.213559 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.213519    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"]
Nov 03 07:28:44.219400 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.219379    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:28:44.338158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.327225    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-acdd08\""
Nov 03 07:28:44.338158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.328195    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-acdd08-dockercfg-dc94s\""
Nov 03 07:28:44.338158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.332493    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"]
Nov 03 07:28:44.350158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.339107    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh8mz\" (UniqueName: \"kubernetes.io/projected/7f3f0492-22c0-457d-8e64-2da01d8c1370-kube-api-access-wh8mz\") pod \"payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd\" (UID: \"7f3f0492-22c0-457d-8e64-2da01d8c1370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:28:44.352324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.351628    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3f0492-22c0-457d-8e64-2da01d8c1370-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd\" (UID: \"7f3f0492-22c0-457d-8e64-2da01d8c1370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:28:44.454787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.453256    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh8mz\" (UniqueName: \"kubernetes.io/projected/7f3f0492-22c0-457d-8e64-2da01d8c1370-kube-api-access-wh8mz\") pod \"payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd\" (UID: \"7f3f0492-22c0-457d-8e64-2da01d8c1370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:28:44.454787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.453422    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3f0492-22c0-457d-8e64-2da01d8c1370-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd\" (UID: \"7f3f0492-22c0-457d-8e64-2da01d8c1370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:28:44.454787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.454448    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3f0492-22c0-457d-8e64-2da01d8c1370-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd\" (UID: \"7f3f0492-22c0-457d-8e64-2da01d8c1370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:28:44.522198 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.522098    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh8mz\" (UniqueName: \"kubernetes.io/projected/7f3f0492-22c0-457d-8e64-2da01d8c1370-kube-api-access-wh8mz\") pod \"payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd\" (UID: \"7f3f0492-22c0-457d-8e64-2da01d8c1370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:28:44.546028 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.545988    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:28:44.953461 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:28:44.953429    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3f0492_22c0_457d_8e64_2da01d8c1370.slice/crio-3e631d982971fef7f5f7899cdaa3d329726c40a095157acebecc23641a7940b6 WatchSource:0}: Error finding container 3e631d982971fef7f5f7899cdaa3d329726c40a095157acebecc23641a7940b6: Status 404 returned error can't find the container with id 3e631d982971fef7f5f7899cdaa3d329726c40a095157acebecc23641a7940b6
Nov 03 07:28:44.954089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:44.954066    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"]
Nov 03 07:28:45.839542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:45.839513    4470 generic.go:356] "Generic (PLEG): container finished" podID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerID="4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f" exitCode=0
Nov 03 07:28:45.840027 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:45.839585    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" event={"ID":"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba","Type":"ContainerDied","Data":"4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f"}
Nov 03 07:28:45.840126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:45.840109    4470 scope.go:122] "RemoveContainer" containerID="4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f"
Nov 03 07:28:45.843535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:45.843043    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:28:45.843535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:45.843235    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:28:45.843535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:45.843334    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:28:45.847846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:45.847817    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd" event={"ID":"7f3f0492-22c0-457d-8e64-2da01d8c1370","Type":"ContainerStarted","Data":"42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131"}
Nov 03 07:28:45.847973 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:45.847903    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd" event={"ID":"7f3f0492-22c0-457d-8e64-2da01d8c1370","Type":"ContainerStarted","Data":"3e631d982971fef7f5f7899cdaa3d329726c40a095157acebecc23641a7940b6"}
Nov 03 07:28:45.848249 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:45.848228    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:28:46.116313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:46.116195    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd" podStartSLOduration=2.116173335 podStartE2EDuration="2.116173335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:46.115073109 +0000 UTC m=+1778.447442104" watchObservedRunningTime="2026-11-03 07:28:46.116173335 +0000 UTC m=+1778.448542330"
Nov 03 07:28:46.992567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:46.992487    4470 generic.go:356] "Generic (PLEG): container finished" podID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerID="db4ca98833575ec82957daba56192550d1b065c5060119843f7596987232395a" exitCode=0
Nov 03 07:28:46.993054 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:46.992672    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" event={"ID":"381d4897-ba0e-4795-86d4-45fc61fdfcb7","Type":"ContainerDied","Data":"db4ca98833575ec82957daba56192550d1b065c5060119843f7596987232395a"}
Nov 03 07:28:46.997237 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:46.997211    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" event={"ID":"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba","Type":"ContainerStarted","Data":"a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1"}
Nov 03 07:28:47.031749 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.031725    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:28:47.171172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.168071    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:28:47.171172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.168240    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 07:28:47.171172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.168302    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 07:28:47.214752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.214718    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-certs\") pod \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") "
Nov 03 07:28:47.214918 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.214792    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-envoy\") pod \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") "
Nov 03 07:28:47.214918 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.214877    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-kube-api-access-c8qpj\" (UniqueName: \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-kube-api-access-c8qpj\") pod \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") "
Nov 03 07:28:47.215029 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.215016    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-data\") pod \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") "
Nov 03 07:28:47.215085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.215027    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-certs" pod "381d4897-ba0e-4795-86d4-45fc61fdfcb7" (UID: "381d4897-ba0e-4795-86d4-45fc61fdfcb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:47.215085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.215067    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-credential-socket\") pod \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") "
Nov 03 07:28:47.215233 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.215144    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-token\" (UniqueName: \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-token\") pod \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") "
Nov 03 07:28:47.215233 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.215173    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istiod-ca-cert\") pod \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") "
Nov 03 07:28:47.215343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.215264    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-podinfo\") pod \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") "
Nov 03 07:28:47.215343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.215295    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-data" pod "381d4897-ba0e-4795-86d4-45fc61fdfcb7" (UID: "381d4897-ba0e-4795-86d4-45fc61fdfcb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:47.215445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.215312    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-socket\") pod \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\" (UID: \"381d4897-ba0e-4795-86d4-45fc61fdfcb7\") "
Nov 03 07:28:47.215445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.215354    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-credential-socket" pod "381d4897-ba0e-4795-86d4-45fc61fdfcb7" (UID: "381d4897-ba0e-4795-86d4-45fc61fdfcb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:47.215625 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.215593    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-socket" pod "381d4897-ba0e-4795-86d4-45fc61fdfcb7" (UID: "381d4897-ba0e-4795-86d4-45fc61fdfcb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:47.215853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.215722    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istiod-ca-cert" pod "381d4897-ba0e-4795-86d4-45fc61fdfcb7" (UID: "381d4897-ba0e-4795-86d4-45fc61fdfcb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:28:47.216043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.216023    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:47.216113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.216050    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:47.216113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.216064    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:47.216113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.216079    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:47.216113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.216093    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:47.217460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.217431    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-envoy" pod "381d4897-ba0e-4795-86d4-45fc61fdfcb7" (UID: "381d4897-ba0e-4795-86d4-45fc61fdfcb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:28:47.217562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.217491    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-kube-api-access-c8qpj" pod "381d4897-ba0e-4795-86d4-45fc61fdfcb7" (UID: "381d4897-ba0e-4795-86d4-45fc61fdfcb7"). InnerVolumeSpecName "kube-api-access-c8qpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:28:47.217961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.217939    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-podinfo" pod "381d4897-ba0e-4795-86d4-45fc61fdfcb7" (UID: "381d4897-ba0e-4795-86d4-45fc61fdfcb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:28:47.218013 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.217952    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-token" pod "381d4897-ba0e-4795-86d4-45fc61fdfcb7" (UID: "381d4897-ba0e-4795-86d4-45fc61fdfcb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:28:47.317918 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.317887    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8qpj\" (UniqueName: \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-kube-api-access-c8qpj\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:47.317918 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.317918    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:47.318126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.317929    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:47.318126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:47.317938    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381d4897-ba0e-4795-86d4-45fc61fdfcb7-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:28:48.016423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:48.016391    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw" event={"ID":"381d4897-ba0e-4795-86d4-45fc61fdfcb7","Type":"ContainerDied","Data":"f1012e3539133080c62326bcd80048addf1c311096062aa1d58e50c9f476946c"}
Nov 03 07:28:48.016876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:48.016449    4470 scope.go:122] "RemoveContainer" containerID="db4ca98833575ec82957daba56192550d1b065c5060119843f7596987232395a"
Nov 03 07:28:48.016876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:48.016469    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"
Nov 03 07:28:48.177239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:48.166539    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 07:28:48.177239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:48.166618    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 07:28:48.199691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:48.199653    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"]
Nov 03 07:28:48.278088 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:48.278046    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw"]
Nov 03 07:28:49.172263 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:49.171295    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 07:28:49.172263 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:49.171353    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 07:28:49.808710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:49.808672    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" path="/var/lib/kubelet/pods/381d4897-ba0e-4795-86d4-45fc61fdfcb7/volumes"
Nov 03 07:28:50.171582 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:50.171543    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 07:28:50.171804 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:50.171613    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 07:28:50.274753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:50.274726    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:28:51.168555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:51.168521    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 07:28:51.168838 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:51.168806    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 07:28:51.784698 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:51.784670    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:28:52.164336 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:52.164304    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 07:28:52.164516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:52.164358    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 07:28:53.169487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:53.169458    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:28:53.170505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:53.170483    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:28:57.001118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:57.001082    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:28:57.944944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:57.944906    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"]
Nov 03 07:28:57.945697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:57.945680    4470 cpu_manager.go:414] "containerMap: removing container" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy"
Nov 03 07:28:57.945756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:57.945700    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy"
Nov 03 07:28:57.945821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:57.945812    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="381d4897-ba0e-4795-86d4-45fc61fdfcb7" containerName="istio-proxy"
Nov 03 07:28:57.950998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:57.950980    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:28:58.009622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.009587    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"]
Nov 03 07:28:58.042790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.042751    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:28:58.043018 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.042996    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:28:58.043072 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.043031    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fmtw\" (UniqueName: \"kubernetes.io/projected/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-kube-api-access-6fmtw\") pod \"maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:28:58.055384 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.055346    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:28:58.055590 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.055404    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:28:58.055590 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.055500    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:28:58.055590 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.055522    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:28:58.144406 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.144376    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:28:58.144574 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.144417    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fmtw\" (UniqueName: \"kubernetes.io/projected/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-kube-api-access-6fmtw\") pod \"maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:28:58.144574 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.144535    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:28:58.145109 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.145091    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:28:58.147208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.147125    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:28:58.219645 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.219562    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fmtw\" (UniqueName: \"kubernetes.io/projected/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-kube-api-access-6fmtw\") pod \"maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:28:58.266032 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.265999    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:28:58.651018 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:58.650992    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"]
Nov 03 07:28:58.651367 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:28:58.651344    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5755f3b0_573e_4751_923c_ccaa3f7d8e8a.slice/crio-70e974fa2399eef9b532fdba57342ceb5ee3ca18cdd814d2a282542e36c21d99 WatchSource:0}: Error finding container 70e974fa2399eef9b532fdba57342ceb5ee3ca18cdd814d2a282542e36c21d99: Status 404 returned error can't find the container with id 70e974fa2399eef9b532fdba57342ceb5ee3ca18cdd814d2a282542e36c21d99
Nov 03 07:28:59.164638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:59.164606    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc" event={"ID":"5755f3b0-573e-4751-923c-ccaa3f7d8e8a","Type":"ContainerStarted","Data":"a8d831105096541b36757ce07a7c773a9ae532f9e91382feb63d41a58a270629"}
Nov 03 07:28:59.165009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:59.164656    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc" event={"ID":"5755f3b0-573e-4751-923c-ccaa3f7d8e8a","Type":"ContainerStarted","Data":"70e974fa2399eef9b532fdba57342ceb5ee3ca18cdd814d2a282542e36c21d99"}
Nov 03 07:28:59.291626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:28:59.291507    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc" podStartSLOduration=2.291492792 podStartE2EDuration="2.291492792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:59.287651659 +0000 UTC m=+1791.620020656" watchObservedRunningTime="2026-11-03 07:28:59.291492792 +0000 UTC m=+1791.623861785"
Nov 03 07:29:00.173885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:00.173848    4470 generic.go:356] "Generic (PLEG): container finished" podID="5755f3b0-573e-4751-923c-ccaa3f7d8e8a" containerID="a8d831105096541b36757ce07a7c773a9ae532f9e91382feb63d41a58a270629" exitCode=0
Nov 03 07:29:00.174304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:00.173928    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc" event={"ID":"5755f3b0-573e-4751-923c-ccaa3f7d8e8a","Type":"ContainerDied","Data":"a8d831105096541b36757ce07a7c773a9ae532f9e91382feb63d41a58a270629"}
Nov 03 07:29:01.339529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:01.339506    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:29:01.393678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:01.393653    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-openshift-ca\") pod \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") "
Nov 03 07:29:01.393807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:01.393760    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-kube-api-access-6fmtw\" (UniqueName: \"kubernetes.io/projected/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-kube-api-access-6fmtw\") pod \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") "
Nov 03 07:29:01.393893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:01.393881    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-xks-ca\") pod \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\" (UID: \"5755f3b0-573e-4751-923c-ccaa3f7d8e8a\") "
Nov 03 07:29:01.394048 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:01.394025    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-openshift-ca" pod "5755f3b0-573e-4751-923c-ccaa3f7d8e8a" (UID: "5755f3b0-573e-4751-923c-ccaa3f7d8e8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:29:01.394500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:01.394477    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:01.396284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:01.396260    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-xks-ca" pod "5755f3b0-573e-4751-923c-ccaa3f7d8e8a" (UID: "5755f3b0-573e-4751-923c-ccaa3f7d8e8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:29:01.396563 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:01.396535    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-kube-api-access-6fmtw" pod "5755f3b0-573e-4751-923c-ccaa3f7d8e8a" (UID: "5755f3b0-573e-4751-923c-ccaa3f7d8e8a"). InnerVolumeSpecName "kube-api-access-6fmtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:29:01.496265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:01.496190    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fmtw\" (UniqueName: \"kubernetes.io/projected/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-kube-api-access-6fmtw\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:01.496265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:01.496221    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5755f3b0-573e-4751-923c-ccaa3f7d8e8a-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:02.191055 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:02.190976    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"
Nov 03 07:29:02.191055 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:02.191013    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc" event={"ID":"5755f3b0-573e-4751-923c-ccaa3f7d8e8a","Type":"ContainerDied","Data":"70e974fa2399eef9b532fdba57342ceb5ee3ca18cdd814d2a282542e36c21d99"}
Nov 03 07:29:02.191393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:02.191057    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70e974fa2399eef9b532fdba57342ceb5ee3ca18cdd814d2a282542e36c21d99"
Nov 03 07:29:04.761223 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:29:04.761186    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:29:09.848741 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:09.848707    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"]
Nov 03 07:29:09.962008 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:09.961976    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"]
Nov 03 07:29:13.363496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.363464    4470 scope.go:122] "RemoveContainer" containerID="af0eb1357cd182620c39180d3ae05561f42dd297fe3182db16e110863623dd71"
Nov 03 07:29:13.374362 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.374342    4470 scope.go:122] "RemoveContainer" containerID="6cbb450e2460ac0d511011074b36a6872f8333ecaebe193adc66b6b4f4c67eca"
Nov 03 07:29:13.383858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.383835    4470 scope.go:122] "RemoveContainer" containerID="7ec1b18769876bc96d15219a3ac3934549ac7d79abd3efadaf01366449b4b98d"
Nov 03 07:29:13.393235 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.393219    4470 scope.go:122] "RemoveContainer" containerID="8d80225475e724b90a673b821569b59020f83a5cbb6e60d1adc4a4e728fde223"
Nov 03 07:29:13.402635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.402613    4470 scope.go:122] "RemoveContainer" containerID="22d9588ec67a6f55d1d111ce8b8d2ce90327f88798e0ea89324cd447ce023218"
Nov 03 07:29:13.411938 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.411922    4470 scope.go:122] "RemoveContainer" containerID="cfc317cc5a93d173ba3f8232259a6aad09ebcabc697803ce82902cdfdf8ddfbc"
Nov 03 07:29:13.421870 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.421849    4470 scope.go:122] "RemoveContainer" containerID="9ebfdf73c522a5c887d4ae6c66c31692bc3ab0a7f2e21de29288bac68704191a"
Nov 03 07:29:13.432984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.432966    4470 scope.go:122] "RemoveContainer" containerID="d051da0d5cc72394c239a14023c574f31c6fb41876c6c26ab6151ad913d06a60"
Nov 03 07:29:13.442280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.442255    4470 scope.go:122] "RemoveContainer" containerID="406a06a9533f46374c554a5ec57805a22b68c21d96ea515540706f0305b2bbbf"
Nov 03 07:29:13.452394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.452375    4470 scope.go:122] "RemoveContainer" containerID="f563e55bfca05660c54a59a8d2d0079e842eb5df28366212067d453f79e829bf"
Nov 03 07:29:13.462194 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.462172    4470 scope.go:122] "RemoveContainer" containerID="1aaa457ea24bafcd23260959694efdc03c1dd1b0f3f56ddb2b2409df05908d9a"
Nov 03 07:29:13.473070 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:13.473051    4470 scope.go:122] "RemoveContainer" containerID="b3fe78465965ee147ee8d9beb8a86360063e069288e6894cd33245bb506826f6"
Nov 03 07:29:14.849268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:14.849200    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh" podUID="afd8a6cd-4714-4d4a-ab14-55e8e0145b45" containerName="payload-processing" containerID="cri-o://7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056" gracePeriod=25
Nov 03 07:29:14.962365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:14.962306    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg" podUID="f3f3adb8-b332-4a67-b345-561ed0da7c99" containerName="payload-pre-processing" containerID="cri-o://15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b" gracePeriod=25
Nov 03 07:29:15.167671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.167644    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:29:15.235740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.235704    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-kube-api-access-zg5w4\" (UniqueName: \"kubernetes.io/projected/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-kube-api-access-zg5w4\") pod \"afd8a6cd-4714-4d4a-ab14-55e8e0145b45\" (UID: \"afd8a6cd-4714-4d4a-ab14-55e8e0145b45\") "
Nov 03 07:29:15.236043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.236024    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-plugins-config-volume\") pod \"afd8a6cd-4714-4d4a-ab14-55e8e0145b45\" (UID: \"afd8a6cd-4714-4d4a-ab14-55e8e0145b45\") "
Nov 03 07:29:15.236515 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.236484    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-plugins-config-volume" pod "afd8a6cd-4714-4d4a-ab14-55e8e0145b45" (UID: "afd8a6cd-4714-4d4a-ab14-55e8e0145b45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:29:15.236913 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.236893    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:15.239165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.239118    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-kube-api-access-zg5w4" pod "afd8a6cd-4714-4d4a-ab14-55e8e0145b45" (UID: "afd8a6cd-4714-4d4a-ab14-55e8e0145b45"). InnerVolumeSpecName "kube-api-access-zg5w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:29:15.266205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.266184    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:29:15.305181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.299437    4470 generic.go:356] "Generic (PLEG): container finished" podID="afd8a6cd-4714-4d4a-ab14-55e8e0145b45" containerID="7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056" exitCode=0
Nov 03 07:29:15.305181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.299473    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh" event={"ID":"afd8a6cd-4714-4d4a-ab14-55e8e0145b45","Type":"ContainerDied","Data":"7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056"}
Nov 03 07:29:15.305181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.299496    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"
Nov 03 07:29:15.305181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.299559    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh" event={"ID":"afd8a6cd-4714-4d4a-ab14-55e8e0145b45","Type":"ContainerDied","Data":"fcbcd867d2d6c4df14539dc5a41fac692198270831ce66d91c2d72ade8d13d6f"}
Nov 03 07:29:15.305181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.299593    4470 scope.go:122] "RemoveContainer" containerID="7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056"
Nov 03 07:29:15.305181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.301319    4470 generic.go:356] "Generic (PLEG): container finished" podID="f3f3adb8-b332-4a67-b345-561ed0da7c99" containerID="15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b" exitCode=0
Nov 03 07:29:15.305181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.301381    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"
Nov 03 07:29:15.305181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.301401    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg" event={"ID":"f3f3adb8-b332-4a67-b345-561ed0da7c99","Type":"ContainerDied","Data":"15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b"}
Nov 03 07:29:15.305181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.301437    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg" event={"ID":"f3f3adb8-b332-4a67-b345-561ed0da7c99","Type":"ContainerDied","Data":"4d1c57fc99d11d70f16244eab7bba8385e79738a9932d2581fb5e43319430fb6"}
Nov 03 07:29:15.311544 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.311526    4470 scope.go:122] "RemoveContainer" containerID="7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056"
Nov 03 07:29:15.311877 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:29:15.311857    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056\": container with ID starting with 7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056 not found: ID does not exist" containerID="7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056"
Nov 03 07:29:15.311948 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.311885    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056"} err="failed to get container status \"7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056\": rpc error: code = NotFound desc = could not find container \"7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056\": container with ID starting with 7c9d2ef9806e1d36289628cfc4cbb0098039aaee724e7c55e49b29985775d056 not found: ID does not exist"
Nov 03 07:29:15.311948 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.311904    4470 scope.go:122] "RemoveContainer" containerID="15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b"
Nov 03 07:29:15.322049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.322022    4470 scope.go:122] "RemoveContainer" containerID="15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b"
Nov 03 07:29:15.322412 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:29:15.322392    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b\": container with ID starting with 15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b not found: ID does not exist" containerID="15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b"
Nov 03 07:29:15.322497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.322418    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b"} err="failed to get container status \"15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b\": rpc error: code = NotFound desc = could not find container \"15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b\": container with ID starting with 15724ff5294e0943d6194ed9ab2fb1c2cbd6d991c908acfa09afb0534620390b not found: ID does not exist"
Nov 03 07:29:15.338126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.338101    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f3adb8-b332-4a67-b345-561ed0da7c99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f3adb8-b332-4a67-b345-561ed0da7c99-plugins-config-volume\") pod \"f3f3adb8-b332-4a67-b345-561ed0da7c99\" (UID: \"f3f3adb8-b332-4a67-b345-561ed0da7c99\") "
Nov 03 07:29:15.338257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.338186    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f3adb8-b332-4a67-b345-561ed0da7c99-kube-api-access-vxvrm\" (UniqueName: \"kubernetes.io/projected/f3f3adb8-b332-4a67-b345-561ed0da7c99-kube-api-access-vxvrm\") pod \"f3f3adb8-b332-4a67-b345-561ed0da7c99\" (UID: \"f3f3adb8-b332-4a67-b345-561ed0da7c99\") "
Nov 03 07:29:15.338583 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.338550    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f3adb8-b332-4a67-b345-561ed0da7c99-plugins-config-volume" pod "f3f3adb8-b332-4a67-b345-561ed0da7c99" (UID: "f3f3adb8-b332-4a67-b345-561ed0da7c99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:29:15.338823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.338804    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg5w4\" (UniqueName: \"kubernetes.io/projected/afd8a6cd-4714-4d4a-ab14-55e8e0145b45-kube-api-access-zg5w4\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:15.338905 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.338829    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f3adb8-b332-4a67-b345-561ed0da7c99-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:15.340637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.340614    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f3adb8-b332-4a67-b345-561ed0da7c99-kube-api-access-vxvrm" pod "f3f3adb8-b332-4a67-b345-561ed0da7c99" (UID: "f3f3adb8-b332-4a67-b345-561ed0da7c99"). InnerVolumeSpecName "kube-api-access-vxvrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:29:15.440729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.440648    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxvrm\" (UniqueName: \"kubernetes.io/projected/f3f3adb8-b332-4a67-b345-561ed0da7c99-kube-api-access-vxvrm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:15.530176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.530128    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"]
Nov 03 07:29:15.535751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.535726    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh"]
Nov 03 07:29:15.729637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.729557    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"]
Nov 03 07:29:15.735199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.735173    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg"]
Nov 03 07:29:15.808248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.808211    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd8a6cd-4714-4d4a-ab14-55e8e0145b45" path="/var/lib/kubelet/pods/afd8a6cd-4714-4d4a-ab14-55e8e0145b45/volumes"
Nov 03 07:29:15.808702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:15.808688    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f3adb8-b332-4a67-b345-561ed0da7c99" path="/var/lib/kubelet/pods/f3f3adb8-b332-4a67-b345-561ed0da7c99/volumes"
Nov 03 07:29:26.957110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:26.957076    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"]
Nov 03 07:29:26.957612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:26.957344    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy" containerID="cri-o://f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d" gracePeriod=30
Nov 03 07:29:27.007565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:27.007532    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"]
Nov 03 07:29:27.007798 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:27.007771    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy" containerID="cri-o://fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2" gracePeriod=30
Nov 03 07:29:32.306844 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.306808    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:29:32.344106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.344079    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:29:32.426925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.426891    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-credential-socket\") pod \"368b3be0-0b24-4b53-a188-4920f1c46113\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") "
Nov 03 07:29:32.427120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.426974    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-istio-token\" (UniqueName: \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-istio-token\") pod \"368b3be0-0b24-4b53-a188-4920f1c46113\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") "
Nov 03 07:29:32.427120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427043    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-envoy\") pod \"368b3be0-0b24-4b53-a188-4920f1c46113\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") "
Nov 03 07:29:32.427120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427075    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/368b3be0-0b24-4b53-a188-4920f1c46113-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/368b3be0-0b24-4b53-a188-4920f1c46113-istio-podinfo\") pod \"368b3be0-0b24-4b53-a188-4920f1c46113\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") "
Nov 03 07:29:32.427346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427164    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/368b3be0-0b24-4b53-a188-4920f1c46113-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/368b3be0-0b24-4b53-a188-4920f1c46113-istiod-ca-cert\") pod \"368b3be0-0b24-4b53-a188-4920f1c46113\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") "
Nov 03 07:29:32.427346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427197    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-certs\") pod \"368b3be0-0b24-4b53-a188-4920f1c46113\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") "
Nov 03 07:29:32.427346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427248    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-credential-socket" pod "368b3be0-0b24-4b53-a188-4920f1c46113" (UID: "368b3be0-0b24-4b53-a188-4920f1c46113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:29:32.427346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427258    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-kube-api-access-8l82g\" (UniqueName: \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-kube-api-access-8l82g\") pod \"368b3be0-0b24-4b53-a188-4920f1c46113\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") "
Nov 03 07:29:32.427555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427345    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-socket\") pod \"40e10d85-28e0-4333-91c0-0bb95e661abe\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") "
Nov 03 07:29:32.427555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427429    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-kube-api-access-lbm2r\" (UniqueName: \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-kube-api-access-lbm2r\") pod \"40e10d85-28e0-4333-91c0-0bb95e661abe\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") "
Nov 03 07:29:32.427555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427519    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-socket\") pod \"368b3be0-0b24-4b53-a188-4920f1c46113\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") "
Nov 03 07:29:32.427744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427577    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-token\" (UniqueName: \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-token\") pod \"40e10d85-28e0-4333-91c0-0bb95e661abe\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") "
Nov 03 07:29:32.427744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427584    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/368b3be0-0b24-4b53-a188-4920f1c46113-istiod-ca-cert" pod "368b3be0-0b24-4b53-a188-4920f1c46113" (UID: "368b3be0-0b24-4b53-a188-4920f1c46113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:29:32.427744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427623    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-certs\") pod \"40e10d85-28e0-4333-91c0-0bb95e661abe\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") "
Nov 03 07:29:32.427744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427693    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-socket" pod "40e10d85-28e0-4333-91c0-0bb95e661abe" (UID: "40e10d85-28e0-4333-91c0-0bb95e661abe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:29:32.427744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427711    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-data\") pod \"40e10d85-28e0-4333-91c0-0bb95e661abe\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") "
Nov 03 07:29:32.428202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427866    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-certs" pod "368b3be0-0b24-4b53-a188-4920f1c46113" (UID: "368b3be0-0b24-4b53-a188-4920f1c46113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:29:32.428202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.427897    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-socket" pod "368b3be0-0b24-4b53-a188-4920f1c46113" (UID: "368b3be0-0b24-4b53-a188-4920f1c46113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:29:32.428202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.428018    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-podinfo\") pod \"40e10d85-28e0-4333-91c0-0bb95e661abe\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") "
Nov 03 07:29:32.428202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.428054    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-credential-socket\") pod \"40e10d85-28e0-4333-91c0-0bb95e661abe\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") "
Nov 03 07:29:32.428202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.428101    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-data\") pod \"368b3be0-0b24-4b53-a188-4920f1c46113\" (UID: \"368b3be0-0b24-4b53-a188-4920f1c46113\") "
Nov 03 07:29:32.428202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.428152    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e10d85-28e0-4333-91c0-0bb95e661abe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e10d85-28e0-4333-91c0-0bb95e661abe-istiod-ca-cert\") pod \"40e10d85-28e0-4333-91c0-0bb95e661abe\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") "
Nov 03 07:29:32.428202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.428182    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-envoy\") pod \"40e10d85-28e0-4333-91c0-0bb95e661abe\" (UID: \"40e10d85-28e0-4333-91c0-0bb95e661abe\") "
Nov 03 07:29:32.428596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.428291    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-certs" pod "40e10d85-28e0-4333-91c0-0bb95e661abe" (UID: "40e10d85-28e0-4333-91c0-0bb95e661abe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:29:32.428596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.428480    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-credential-socket" pod "40e10d85-28e0-4333-91c0-0bb95e661abe" (UID: "40e10d85-28e0-4333-91c0-0bb95e661abe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:29:32.429208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.429010    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.429208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.429039    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/368b3be0-0b24-4b53-a188-4920f1c46113-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.429208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.429053    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.429208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.429066    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.429208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.429082    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.429208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.429096    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.429208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.429110    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.430318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.429359    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-data" pod "368b3be0-0b24-4b53-a188-4920f1c46113" (UID: "368b3be0-0b24-4b53-a188-4920f1c46113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:29:32.430318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.429483    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-data" pod "40e10d85-28e0-4333-91c0-0bb95e661abe" (UID: "40e10d85-28e0-4333-91c0-0bb95e661abe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:29:32.430318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.429659    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e10d85-28e0-4333-91c0-0bb95e661abe-istiod-ca-cert" pod "40e10d85-28e0-4333-91c0-0bb95e661abe" (UID: "40e10d85-28e0-4333-91c0-0bb95e661abe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:29:32.430536 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.430513    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-envoy" pod "368b3be0-0b24-4b53-a188-4920f1c46113" (UID: "368b3be0-0b24-4b53-a188-4920f1c46113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:29:32.430536 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.430522    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/368b3be0-0b24-4b53-a188-4920f1c46113-istio-podinfo" pod "368b3be0-0b24-4b53-a188-4920f1c46113" (UID: "368b3be0-0b24-4b53-a188-4920f1c46113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:29:32.430641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.430588    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-kube-api-access-8l82g" pod "368b3be0-0b24-4b53-a188-4920f1c46113" (UID: "368b3be0-0b24-4b53-a188-4920f1c46113"). InnerVolumeSpecName "kube-api-access-8l82g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:29:32.430641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.430627    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-istio-token" pod "368b3be0-0b24-4b53-a188-4920f1c46113" (UID: "368b3be0-0b24-4b53-a188-4920f1c46113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:29:32.430816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.430786    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-envoy" pod "40e10d85-28e0-4333-91c0-0bb95e661abe" (UID: "40e10d85-28e0-4333-91c0-0bb95e661abe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:29:32.431243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.431222    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-kube-api-access-lbm2r" pod "40e10d85-28e0-4333-91c0-0bb95e661abe" (UID: "40e10d85-28e0-4333-91c0-0bb95e661abe"). InnerVolumeSpecName "kube-api-access-lbm2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:29:32.431491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.431477    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-token" pod "40e10d85-28e0-4333-91c0-0bb95e661abe" (UID: "40e10d85-28e0-4333-91c0-0bb95e661abe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:29:32.432212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.432191    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-podinfo" pod "40e10d85-28e0-4333-91c0-0bb95e661abe" (UID: "40e10d85-28e0-4333-91c0-0bb95e661abe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:29:32.459520 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.459485    4470 generic.go:356] "Generic (PLEG): container finished" podID="368b3be0-0b24-4b53-a188-4920f1c46113" containerID="f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d" exitCode=0
Nov 03 07:29:32.459678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.459529    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"
Nov 03 07:29:32.459678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.459545    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" event={"ID":"368b3be0-0b24-4b53-a188-4920f1c46113","Type":"ContainerDied","Data":"f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d"}
Nov 03 07:29:32.459678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.459597    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m" event={"ID":"368b3be0-0b24-4b53-a188-4920f1c46113","Type":"ContainerDied","Data":"71493da0631e2e834e1f481d77b744d1dcd8e63bb4aa4c9bc74e7b720b43fa50"}
Nov 03 07:29:32.459678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.459629    4470 scope.go:122] "RemoveContainer" containerID="f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d"
Nov 03 07:29:32.461454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.461425    4470 generic.go:356] "Generic (PLEG): container finished" podID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerID="fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2" exitCode=0
Nov 03 07:29:32.461549 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.461472    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" event={"ID":"40e10d85-28e0-4333-91c0-0bb95e661abe","Type":"ContainerDied","Data":"fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2"}
Nov 03 07:29:32.461549 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.461512    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc" event={"ID":"40e10d85-28e0-4333-91c0-0bb95e661abe","Type":"ContainerDied","Data":"973e755ccd604bbf41a2fc2e980a5abee4b386f2fe09ca570252022adeed5396"}
Nov 03 07:29:32.461634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.461577    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"
Nov 03 07:29:32.478427 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.478406    4470 scope.go:122] "RemoveContainer" containerID="f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d"
Nov 03 07:29:32.479078 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:29:32.479033    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d\": container with ID starting with f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d not found: ID does not exist" containerID="f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d"
Nov 03 07:29:32.479223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.479085    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d"} err="failed to get container status \"f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d\": rpc error: code = NotFound desc = could not find container \"f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d\": container with ID starting with f2e7b9cacfa2826edf8e8f93287c9fc5b8d630806a06e13ba8283a1e75fe830d not found: ID does not exist"
Nov 03 07:29:32.479223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.479109    4470 scope.go:122] "RemoveContainer" containerID="fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2"
Nov 03 07:29:32.494387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.494357    4470 scope.go:122] "RemoveContainer" containerID="fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2"
Nov 03 07:29:32.494785 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:29:32.494745    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2\": container with ID starting with fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2 not found: ID does not exist" containerID="fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2"
Nov 03 07:29:32.494881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.494784    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2"} err="failed to get container status \"fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2\": rpc error: code = NotFound desc = could not find container \"fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2\": container with ID starting with fe5e59bd7d9ed41b174ae405f4acc019bd574602bc2a0be24bbe357bcea1d7f2 not found: ID does not exist"
Nov 03 07:29:32.530513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.530458    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.530513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.530508    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.530730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.530525    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.530730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.530541    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.530730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.530555    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e10d85-28e0-4333-91c0-0bb95e661abe-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.530730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.530568    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e10d85-28e0-4333-91c0-0bb95e661abe-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.530730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.530582    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.530730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.530596    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/368b3be0-0b24-4b53-a188-4920f1c46113-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.530730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.530609    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/368b3be0-0b24-4b53-a188-4920f1c46113-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.530730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.530623    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l82g\" (UniqueName: \"kubernetes.io/projected/368b3be0-0b24-4b53-a188-4920f1c46113-kube-api-access-8l82g\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.530730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.530639    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbm2r\" (UniqueName: \"kubernetes.io/projected/40e10d85-28e0-4333-91c0-0bb95e661abe-kube-api-access-lbm2r\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:32.682044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.681399    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"]
Nov 03 07:29:32.687205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.687177    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc"]
Nov 03 07:29:32.805308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.805227    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"]
Nov 03 07:29:32.808962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:32.808938    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m"]
Nov 03 07:29:33.808319 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:33.808285    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" path="/var/lib/kubelet/pods/368b3be0-0b24-4b53-a188-4920f1c46113/volumes"
Nov 03 07:29:33.808867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:33.808851    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" path="/var/lib/kubelet/pods/40e10d85-28e0-4333-91c0-0bb95e661abe/volumes"
Nov 03 07:29:34.761722 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:29:34.761687    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:29:37.085163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.085117    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"]
Nov 03 07:29:37.085825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.085809    4470 cpu_manager.go:414] "containerMap: removing container" podUID="f3f3adb8-b332-4a67-b345-561ed0da7c99" containerName="payload-pre-processing"
Nov 03 07:29:37.085875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.085826    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f3adb8-b332-4a67-b345-561ed0da7c99" containerName="payload-pre-processing"
Nov 03 07:29:37.085875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.085851    4470 cpu_manager.go:414] "containerMap: removing container" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy"
Nov 03 07:29:37.085875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.085856    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy"
Nov 03 07:29:37.085965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.085876    4470 cpu_manager.go:414] "containerMap: removing container" podUID="afd8a6cd-4714-4d4a-ab14-55e8e0145b45" containerName="payload-processing"
Nov 03 07:29:37.085965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.085882    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd8a6cd-4714-4d4a-ab14-55e8e0145b45" containerName="payload-processing"
Nov 03 07:29:37.085965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.085892    4470 cpu_manager.go:414] "containerMap: removing container" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy"
Nov 03 07:29:37.085965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.085897    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy"
Nov 03 07:29:37.085965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.085919    4470 cpu_manager.go:414] "containerMap: removing container" podUID="5755f3b0-573e-4751-923c-ccaa3f7d8e8a" containerName="revoke-keys"
Nov 03 07:29:37.085965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.085924    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5755f3b0-573e-4751-923c-ccaa3f7d8e8a" containerName="revoke-keys"
Nov 03 07:29:37.086171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.086018    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f3adb8-b332-4a67-b345-561ed0da7c99" containerName="payload-pre-processing"
Nov 03 07:29:37.086171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.086027    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e10d85-28e0-4333-91c0-0bb95e661abe" containerName="istio-proxy"
Nov 03 07:29:37.086171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.086035    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd8a6cd-4714-4d4a-ab14-55e8e0145b45" containerName="payload-processing"
Nov 03 07:29:37.086171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.086042    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="368b3be0-0b24-4b53-a188-4920f1c46113" containerName="istio-proxy"
Nov 03 07:29:37.086171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.086051    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="5755f3b0-573e-4751-923c-ccaa3f7d8e8a" containerName="revoke-keys"
Nov 03 07:29:37.091271 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.091252    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:37.149117 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.149087    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"]
Nov 03 07:29:37.195177 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.195145    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:29:37.195418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.195385    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:29:37.195499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.195415    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:29:37.196487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.196468    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:29:37.209327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.209301    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6v45\" (UniqueName: \"kubernetes.io/projected/764a6883-b3a7-4efb-a411-9fd14e758a6a-kube-api-access-f6v45\") pod \"maas-api-revoke-sub-9f56d732-h459g\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:37.209512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.209495    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f56d732-h459g\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:37.209634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.209619    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f56d732-h459g\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:37.311408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.311373    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f56d732-h459g\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:37.311574 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.311512    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6v45\" (UniqueName: \"kubernetes.io/projected/764a6883-b3a7-4efb-a411-9fd14e758a6a-kube-api-access-f6v45\") pod \"maas-api-revoke-sub-9f56d732-h459g\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:37.311719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.311699    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f56d732-h459g\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:37.312142 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.312101    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f56d732-h459g\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:37.314126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.314105    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f56d732-h459g\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:37.394845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.394765    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6v45\" (UniqueName: \"kubernetes.io/projected/764a6883-b3a7-4efb-a411-9fd14e758a6a-kube-api-access-f6v45\") pod \"maas-api-revoke-sub-9f56d732-h459g\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:37.405704 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.405664    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:37.796030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:37.796003    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"]
Nov 03 07:29:37.796937 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:29:37.796894    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764a6883_b3a7_4efb_a411_9fd14e758a6a.slice/crio-b201cdca1c0c8aff8cbde35ac4535025fe6b19ce9df9ca668c171374c9f0fbaf WatchSource:0}: Error finding container b201cdca1c0c8aff8cbde35ac4535025fe6b19ce9df9ca668c171374c9f0fbaf: Status 404 returned error can't find the container with id b201cdca1c0c8aff8cbde35ac4535025fe6b19ce9df9ca668c171374c9f0fbaf
Nov 03 07:29:38.521354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:38.521316    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g" event={"ID":"764a6883-b3a7-4efb-a411-9fd14e758a6a","Type":"ContainerStarted","Data":"b54b4569e09e50310a941394c08d862cb4856bda44b65c4b51c76a5b1674f164"}
Nov 03 07:29:38.521777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:38.521411    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g" event={"ID":"764a6883-b3a7-4efb-a411-9fd14e758a6a","Type":"ContainerStarted","Data":"b201cdca1c0c8aff8cbde35ac4535025fe6b19ce9df9ca668c171374c9f0fbaf"}
Nov 03 07:29:38.644472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:38.644411    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g" podStartSLOduration=1.6443967320000001 podStartE2EDuration="1.644396732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:29:38.640830779 +0000 UTC m=+1830.973199767" watchObservedRunningTime="2026-11-03 07:29:38.644396732 +0000 UTC m=+1830.976765755"
Nov 03 07:29:39.529441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:39.529407    4470 generic.go:356] "Generic (PLEG): container finished" podID="764a6883-b3a7-4efb-a411-9fd14e758a6a" containerID="b54b4569e09e50310a941394c08d862cb4856bda44b65c4b51c76a5b1674f164" exitCode=0
Nov 03 07:29:39.529852 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:39.529495    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g" event={"ID":"764a6883-b3a7-4efb-a411-9fd14e758a6a","Type":"ContainerDied","Data":"b54b4569e09e50310a941394c08d862cb4856bda44b65c4b51c76a5b1674f164"}
Nov 03 07:29:40.702321 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:40.702279    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:40.874085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:40.874057    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-xks-ca\") pod \"764a6883-b3a7-4efb-a411-9fd14e758a6a\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") "
Nov 03 07:29:40.874326 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:40.874159    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-openshift-ca\") pod \"764a6883-b3a7-4efb-a411-9fd14e758a6a\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") "
Nov 03 07:29:40.874326 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:40.874225    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764a6883-b3a7-4efb-a411-9fd14e758a6a-kube-api-access-f6v45\" (UniqueName: \"kubernetes.io/projected/764a6883-b3a7-4efb-a411-9fd14e758a6a-kube-api-access-f6v45\") pod \"764a6883-b3a7-4efb-a411-9fd14e758a6a\" (UID: \"764a6883-b3a7-4efb-a411-9fd14e758a6a\") "
Nov 03 07:29:40.874653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:40.874622    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-openshift-ca" pod "764a6883-b3a7-4efb-a411-9fd14e758a6a" (UID: "764a6883-b3a7-4efb-a411-9fd14e758a6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:29:40.875418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:40.875388    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:40.876515 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:40.876489    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-xks-ca" pod "764a6883-b3a7-4efb-a411-9fd14e758a6a" (UID: "764a6883-b3a7-4efb-a411-9fd14e758a6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:29:40.876622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:40.876598    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764a6883-b3a7-4efb-a411-9fd14e758a6a-kube-api-access-f6v45" pod "764a6883-b3a7-4efb-a411-9fd14e758a6a" (UID: "764a6883-b3a7-4efb-a411-9fd14e758a6a"). InnerVolumeSpecName "kube-api-access-f6v45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:29:40.977122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:40.977071    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764a6883-b3a7-4efb-a411-9fd14e758a6a-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:40.977122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:40.977102    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6v45\" (UniqueName: \"kubernetes.io/projected/764a6883-b3a7-4efb-a411-9fd14e758a6a-kube-api-access-f6v45\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:29:41.545226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:41.545193    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g" event={"ID":"764a6883-b3a7-4efb-a411-9fd14e758a6a","Type":"ContainerDied","Data":"b201cdca1c0c8aff8cbde35ac4535025fe6b19ce9df9ca668c171374c9f0fbaf"}
Nov 03 07:29:41.545403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:41.545253    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b201cdca1c0c8aff8cbde35ac4535025fe6b19ce9df9ca668c171374c9f0fbaf"
Nov 03 07:29:41.545403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:41.545200    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"
Nov 03 07:29:49.663680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:49.663595    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"]
Nov 03 07:29:49.711779 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:49.711745    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-399d09e5-n4zgc"]
Nov 03 07:29:49.809034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:49.808995    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5755f3b0-573e-4751-923c-ccaa3f7d8e8a" path="/var/lib/kubelet/pods/5755f3b0-573e-4751-923c-ccaa3f7d8e8a/volumes"
Nov 03 07:29:55.091329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:55.091290    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"]
Nov 03 07:29:55.100571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:55.100537    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87b8bc5f-c5qmh"]
Nov 03 07:29:55.815960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:55.815921    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbeeaa3d-4c6c-4324-8aba-2585e7518b62" path="/var/lib/kubelet/pods/fbeeaa3d-4c6c-4324-8aba-2585e7518b62/volumes"
Nov 03 07:29:57.827790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:57.819094    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:29:57.833318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:57.833222    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:29:57.843608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:57.843477    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:29:57.882278 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:57.882229    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:29:57.955160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:57.898240    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:29:57.955160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:57.927219    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": context deadline exceeded"
Nov 03 07:29:57.955160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:57.927312    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:29:57.955699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:57.955591    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:29:58.004450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:58.004266    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:29:58.004450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:58.004401    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:29:59.066305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:59.065962    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:29:59.066823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:59.066351    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:29:59.066823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:59.066624    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Nov 03 07:29:59.066942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:59.066919    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:29:59.067003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:59.066979    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:29:59.067069 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:59.067018    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:29:59.109276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:59.109245    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:29:59.133225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:59.133002    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc"
Nov 03 07:29:59.215871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:29:59.215699    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:30:00.181871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.181831    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"]
Nov 03 07:30:00.182653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.182635    4470 cpu_manager.go:414] "containerMap: removing container" podUID="764a6883-b3a7-4efb-a411-9fd14e758a6a" containerName="revoke-keys"
Nov 03 07:30:00.182703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.182655    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764a6883-b3a7-4efb-a411-9fd14e758a6a" containerName="revoke-keys"
Nov 03 07:30:00.182815 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.182805    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="764a6883-b3a7-4efb-a411-9fd14e758a6a" containerName="revoke-keys"
Nov 03 07:30:00.186349 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.186324    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"]
Nov 03 07:30:00.186514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.186468    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"
Nov 03 07:30:00.202209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.202179    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"]
Nov 03 07:30:00.202496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.202351    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"
Nov 03 07:30:00.208910 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.208886    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"]
Nov 03 07:30:00.209059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.208997    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"
Nov 03 07:30:00.213519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.213495    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"]
Nov 03 07:30:00.213844 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.213761    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"
Nov 03 07:30:00.215813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.215785    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8tl7\" (UniqueName: \"kubernetes.io/projected/d32f79ba-6fb3-447b-b224-639f835f6d69-kube-api-access-s8tl7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6\" (UID: \"d32f79ba-6fb3-447b-b224-639f835f6d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"
Nov 03 07:30:00.215972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.215948    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07659463-2d5f-4171-9e7b-d3b83cd4b838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2\" (UID: \"07659463-2d5f-4171-9e7b-d3b83cd4b838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"
Nov 03 07:30:00.216038 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.215985    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h562r\" (UniqueName: \"kubernetes.io/projected/07659463-2d5f-4171-9e7b-d3b83cd4b838-kube-api-access-h562r\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2\" (UID: \"07659463-2d5f-4171-9e7b-d3b83cd4b838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"
Nov 03 07:30:00.216234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.216116    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcw5m\" (UniqueName: \"kubernetes.io/projected/dae496e2-15ee-4ac2-8de0-b61d610f0150-kube-api-access-wcw5m\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf\" (UID: \"dae496e2-15ee-4ac2-8de0-b61d610f0150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"
Nov 03 07:30:00.216349 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.216244    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e505ebba-8ed9-4532-988d-5d3ff724b522-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs\" (UID: \"e505ebba-8ed9-4532-988d-5d3ff724b522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"
Nov 03 07:30:00.216349 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.216270    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4sbl\" (UniqueName: \"kubernetes.io/projected/e505ebba-8ed9-4532-988d-5d3ff724b522-kube-api-access-f4sbl\") pod \"maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs\" (UID: \"e505ebba-8ed9-4532-988d-5d3ff724b522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"
Nov 03 07:30:00.216477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.216409    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32f79ba-6fb3-447b-b224-639f835f6d69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6\" (UID: \"d32f79ba-6fb3-447b-b224-639f835f6d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"
Nov 03 07:30:00.216548 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.216516    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae496e2-15ee-4ac2-8de0-b61d610f0150-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf\" (UID: \"dae496e2-15ee-4ac2-8de0-b61d610f0150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"
Nov 03 07:30:00.222331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.222309    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"]
Nov 03 07:30:00.222493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.222437    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"
Nov 03 07:30:00.227865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.227836    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"]
Nov 03 07:30:00.227995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.227940    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"
Nov 03 07:30:00.233427 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.233399    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"
Nov 03 07:30:00.237567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.237542    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"]
Nov 03 07:30:00.242203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.242122    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"
Nov 03 07:30:00.265398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.262985    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:30:00.265398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.263109    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:30:00.265398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.263217    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:30:00.308319 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.308239    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"]
Nov 03 07:30:00.328803 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.328770    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32f79ba-6fb3-447b-b224-639f835f6d69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6\" (UID: \"d32f79ba-6fb3-447b-b224-639f835f6d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"
Nov 03 07:30:00.329095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.329071    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7a5180-e91f-4bec-815a-a870c63c3992-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894850-56g2c\" (UID: \"de7a5180-e91f-4bec-815a-a870c63c3992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"
Nov 03 07:30:00.330509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.330482    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9ntx\" (UniqueName: \"kubernetes.io/projected/bf996b97-74f9-450d-9ca9-826560308eb2-kube-api-access-g9ntx\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8\" (UID: \"bf996b97-74f9-450d-9ca9-826560308eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"
Nov 03 07:30:00.330807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.330765    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae496e2-15ee-4ac2-8de0-b61d610f0150-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf\" (UID: \"dae496e2-15ee-4ac2-8de0-b61d610f0150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"
Nov 03 07:30:00.331056 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.331017    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf996b97-74f9-450d-9ca9-826560308eb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8\" (UID: \"bf996b97-74f9-450d-9ca9-826560308eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"
Nov 03 07:30:00.331242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.331226    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8tl7\" (UniqueName: \"kubernetes.io/projected/d32f79ba-6fb3-447b-b224-639f835f6d69-kube-api-access-s8tl7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6\" (UID: \"d32f79ba-6fb3-447b-b224-639f835f6d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"
Nov 03 07:30:00.331533 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.331500    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3cb7b51-48dc-4368-98a3-640982b6e68e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw\" (UID: \"e3cb7b51-48dc-4368-98a3-640982b6e68e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"
Nov 03 07:30:00.331731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.331712    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxxbl\" (UniqueName: \"kubernetes.io/projected/10559d66-8937-43fe-9f78-953042895f8a-kube-api-access-xxxbl\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2\" (UID: \"10559d66-8937-43fe-9f78-953042895f8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"
Nov 03 07:30:00.331982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.331964    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b9lz\" (UniqueName: \"kubernetes.io/projected/e3cb7b51-48dc-4368-98a3-640982b6e68e-kube-api-access-5b9lz\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw\" (UID: \"e3cb7b51-48dc-4368-98a3-640982b6e68e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"
Nov 03 07:30:00.332200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.332183    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07659463-2d5f-4171-9e7b-d3b83cd4b838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2\" (UID: \"07659463-2d5f-4171-9e7b-d3b83cd4b838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"
Nov 03 07:30:00.332375 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.332359    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h562r\" (UniqueName: \"kubernetes.io/projected/07659463-2d5f-4171-9e7b-d3b83cd4b838-kube-api-access-h562r\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2\" (UID: \"07659463-2d5f-4171-9e7b-d3b83cd4b838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"
Nov 03 07:30:00.332582 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.332563    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qppx\" (UniqueName: \"kubernetes.io/projected/de7a5180-e91f-4bec-815a-a870c63c3992-kube-api-access-5qppx\") pod \"maas-api-key-cleanup-29894850-56g2c\" (UID: \"de7a5180-e91f-4bec-815a-a870c63c3992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"
Nov 03 07:30:00.332825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.332766    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcw5m\" (UniqueName: \"kubernetes.io/projected/dae496e2-15ee-4ac2-8de0-b61d610f0150-kube-api-access-wcw5m\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf\" (UID: \"dae496e2-15ee-4ac2-8de0-b61d610f0150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"
Nov 03 07:30:00.333013 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.332997    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e505ebba-8ed9-4532-988d-5d3ff724b522-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs\" (UID: \"e505ebba-8ed9-4532-988d-5d3ff724b522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"
Nov 03 07:30:00.333176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.333159    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4sbl\" (UniqueName: \"kubernetes.io/projected/e505ebba-8ed9-4532-988d-5d3ff724b522-kube-api-access-f4sbl\") pod \"maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs\" (UID: \"e505ebba-8ed9-4532-988d-5d3ff724b522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"
Nov 03 07:30:00.333410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.333384    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"
Nov 03 07:30:00.334168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.334145    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e505ebba-8ed9-4532-988d-5d3ff724b522-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs\" (UID: \"e505ebba-8ed9-4532-988d-5d3ff724b522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"
Nov 03 07:30:00.334337 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.334315    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10559d66-8937-43fe-9f78-953042895f8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2\" (UID: \"10559d66-8937-43fe-9f78-953042895f8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"
Nov 03 07:30:00.335760 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.335738    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07659463-2d5f-4171-9e7b-d3b83cd4b838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2\" (UID: \"07659463-2d5f-4171-9e7b-d3b83cd4b838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"
Nov 03 07:30:00.337235 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.335981    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae496e2-15ee-4ac2-8de0-b61d610f0150-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf\" (UID: \"dae496e2-15ee-4ac2-8de0-b61d610f0150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"
Nov 03 07:30:00.337333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.337261    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"]
Nov 03 07:30:00.338372 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.338350    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32f79ba-6fb3-447b-b224-639f835f6d69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6\" (UID: \"d32f79ba-6fb3-447b-b224-639f835f6d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"
Nov 03 07:30:00.385156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.385058    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"]
Nov 03 07:30:00.390190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.390162    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"]
Nov 03 07:30:00.396304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.395018    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"]
Nov 03 07:30:00.402573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.402543    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4sbl\" (UniqueName: \"kubernetes.io/projected/e505ebba-8ed9-4532-988d-5d3ff724b522-kube-api-access-f4sbl\") pod \"maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs\" (UID: \"e505ebba-8ed9-4532-988d-5d3ff724b522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"
Nov 03 07:30:00.402805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.402785    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"]
Nov 03 07:30:00.404733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.404682    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcw5m\" (UniqueName: \"kubernetes.io/projected/dae496e2-15ee-4ac2-8de0-b61d610f0150-kube-api-access-wcw5m\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf\" (UID: \"dae496e2-15ee-4ac2-8de0-b61d610f0150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"
Nov 03 07:30:00.404916 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.404898    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8tl7\" (UniqueName: \"kubernetes.io/projected/d32f79ba-6fb3-447b-b224-639f835f6d69-kube-api-access-s8tl7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6\" (UID: \"d32f79ba-6fb3-447b-b224-639f835f6d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"
Nov 03 07:30:00.405472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.405453    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"]
Nov 03 07:30:00.406567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.406545    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h562r\" (UniqueName: \"kubernetes.io/projected/07659463-2d5f-4171-9e7b-d3b83cd4b838-kube-api-access-h562r\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2\" (UID: \"07659463-2d5f-4171-9e7b-d3b83cd4b838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"
Nov 03 07:30:00.414228 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.411497    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"]
Nov 03 07:30:00.431030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.429249    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"]
Nov 03 07:30:00.438205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.437014    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf996b97-74f9-450d-9ca9-826560308eb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8\" (UID: \"bf996b97-74f9-450d-9ca9-826560308eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"
Nov 03 07:30:00.438205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.437192    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crsnp\" (UniqueName: \"kubernetes.io/projected/8e89e6f9-f776-4d44-9643-23c2d7e2922b-kube-api-access-crsnp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb\" (UID: \"8e89e6f9-f776-4d44-9643-23c2d7e2922b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"
Nov 03 07:30:00.438205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.437236    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3cb7b51-48dc-4368-98a3-640982b6e68e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw\" (UID: \"e3cb7b51-48dc-4368-98a3-640982b6e68e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"
Nov 03 07:30:00.438205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.437269    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxxbl\" (UniqueName: \"kubernetes.io/projected/10559d66-8937-43fe-9f78-953042895f8a-kube-api-access-xxxbl\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2\" (UID: \"10559d66-8937-43fe-9f78-953042895f8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"
Nov 03 07:30:00.438205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.437308    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b9lz\" (UniqueName: \"kubernetes.io/projected/e3cb7b51-48dc-4368-98a3-640982b6e68e-kube-api-access-5b9lz\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw\" (UID: \"e3cb7b51-48dc-4368-98a3-640982b6e68e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"
Nov 03 07:30:00.438205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.437386    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qppx\" (UniqueName: \"kubernetes.io/projected/de7a5180-e91f-4bec-815a-a870c63c3992-kube-api-access-5qppx\") pod \"maas-api-key-cleanup-29894850-56g2c\" (UID: \"de7a5180-e91f-4bec-815a-a870c63c3992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"
Nov 03 07:30:00.438205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.437447    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e89e6f9-f776-4d44-9643-23c2d7e2922b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb\" (UID: \"8e89e6f9-f776-4d44-9643-23c2d7e2922b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"
Nov 03 07:30:00.438205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.437478    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10559d66-8937-43fe-9f78-953042895f8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2\" (UID: \"10559d66-8937-43fe-9f78-953042895f8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"
Nov 03 07:30:00.438205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.437618    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7a5180-e91f-4bec-815a-a870c63c3992-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894850-56g2c\" (UID: \"de7a5180-e91f-4bec-815a-a870c63c3992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"
Nov 03 07:30:00.438205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.437661    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9ntx\" (UniqueName: \"kubernetes.io/projected/bf996b97-74f9-450d-9ca9-826560308eb2-kube-api-access-g9ntx\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8\" (UID: \"bf996b97-74f9-450d-9ca9-826560308eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"
Nov 03 07:30:00.438843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.438813    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf996b97-74f9-450d-9ca9-826560308eb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8\" (UID: \"bf996b97-74f9-450d-9ca9-826560308eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"
Nov 03 07:30:00.438941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.438921    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"]
Nov 03 07:30:00.439564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.439535    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3cb7b51-48dc-4368-98a3-640982b6e68e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw\" (UID: \"e3cb7b51-48dc-4368-98a3-640982b6e68e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"
Nov 03 07:30:00.440176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.440126    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10559d66-8937-43fe-9f78-953042895f8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2\" (UID: \"10559d66-8937-43fe-9f78-953042895f8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"
Nov 03 07:30:00.440327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.440202    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7a5180-e91f-4bec-815a-a870c63c3992-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894850-56g2c\" (UID: \"de7a5180-e91f-4bec-815a-a870c63c3992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"
Nov 03 07:30:00.502725 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.502687    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9ntx\" (UniqueName: \"kubernetes.io/projected/bf996b97-74f9-450d-9ca9-826560308eb2-kube-api-access-g9ntx\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8\" (UID: \"bf996b97-74f9-450d-9ca9-826560308eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"
Nov 03 07:30:00.503442 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.503411    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxxbl\" (UniqueName: \"kubernetes.io/projected/10559d66-8937-43fe-9f78-953042895f8a-kube-api-access-xxxbl\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2\" (UID: \"10559d66-8937-43fe-9f78-953042895f8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"
Nov 03 07:30:00.503442 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.503437    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qppx\" (UniqueName: \"kubernetes.io/projected/de7a5180-e91f-4bec-815a-a870c63c3992-kube-api-access-5qppx\") pod \"maas-api-key-cleanup-29894850-56g2c\" (UID: \"de7a5180-e91f-4bec-815a-a870c63c3992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"
Nov 03 07:30:00.503602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.503475    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b9lz\" (UniqueName: \"kubernetes.io/projected/e3cb7b51-48dc-4368-98a3-640982b6e68e-kube-api-access-5b9lz\") pod \"maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw\" (UID: \"e3cb7b51-48dc-4368-98a3-640982b6e68e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"
Nov 03 07:30:00.508834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.508805    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"
Nov 03 07:30:00.523714 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.523689    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"
Nov 03 07:30:00.533584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.533556    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"
Nov 03 07:30:00.539309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.538871    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e89e6f9-f776-4d44-9643-23c2d7e2922b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb\" (UID: \"8e89e6f9-f776-4d44-9643-23c2d7e2922b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"
Nov 03 07:30:00.539309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.539191    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crsnp\" (UniqueName: \"kubernetes.io/projected/8e89e6f9-f776-4d44-9643-23c2d7e2922b-kube-api-access-crsnp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb\" (UID: \"8e89e6f9-f776-4d44-9643-23c2d7e2922b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"
Nov 03 07:30:00.539727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.539697    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e89e6f9-f776-4d44-9643-23c2d7e2922b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb\" (UID: \"8e89e6f9-f776-4d44-9643-23c2d7e2922b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"
Nov 03 07:30:00.544010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.543982    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"
Nov 03 07:30:00.608223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.607598    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"
Nov 03 07:30:00.612912 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.612745    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crsnp\" (UniqueName: \"kubernetes.io/projected/8e89e6f9-f776-4d44-9643-23c2d7e2922b-kube-api-access-crsnp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb\" (UID: \"8e89e6f9-f776-4d44-9643-23c2d7e2922b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"
Nov 03 07:30:00.702688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.701409    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"
Nov 03 07:30:00.702862 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.702691    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"
Nov 03 07:30:00.702862 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.702805    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"
Nov 03 07:30:00.788364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:00.782492    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"
Nov 03 07:30:01.424025 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:01.409775    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"]
Nov 03 07:30:01.431393 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:30:01.431340    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode505ebba_8ed9_4532_988d_5d3ff724b522.slice/crio-aae0a8e671a9ae2adef6e5d32c6b1f86262eca37c045517b82e68ed5d1eb2fb5 WatchSource:0}: Error finding container aae0a8e671a9ae2adef6e5d32c6b1f86262eca37c045517b82e68ed5d1eb2fb5: Status 404 returned error can't find the container with id aae0a8e671a9ae2adef6e5d32c6b1f86262eca37c045517b82e68ed5d1eb2fb5
Nov 03 07:30:01.785949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:01.783905    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"]
Nov 03 07:30:01.841186 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:01.841154    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"]
Nov 03 07:30:01.885621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:01.885586    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"]
Nov 03 07:30:01.888670 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:30:01.888628    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae496e2_15ee_4ac2_8de0_b61d610f0150.slice/crio-800df364d6d1c0fc62fb275d92df8ec7b3bfcdc964a9048f56a1af4a1c036a0e WatchSource:0}: Error finding container 800df364d6d1c0fc62fb275d92df8ec7b3bfcdc964a9048f56a1af4a1c036a0e: Status 404 returned error can't find the container with id 800df364d6d1c0fc62fb275d92df8ec7b3bfcdc964a9048f56a1af4a1c036a0e
Nov 03 07:30:01.889622 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:30:01.889597    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf996b97_74f9_450d_9ca9_826560308eb2.slice/crio-1fcd935ae83629c56895c3cd72dc05614e4634a4f2e7d82bbcb3d278f464dde1 WatchSource:0}: Error finding container 1fcd935ae83629c56895c3cd72dc05614e4634a4f2e7d82bbcb3d278f464dde1: Status 404 returned error can't find the container with id 1fcd935ae83629c56895c3cd72dc05614e4634a4f2e7d82bbcb3d278f464dde1
Nov 03 07:30:01.900223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:01.900194    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"]
Nov 03 07:30:01.977224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:01.976649    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"]
Nov 03 07:30:01.982298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:01.982274    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"]
Nov 03 07:30:02.008589 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.008562    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"]
Nov 03 07:30:02.012692 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.012666    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"]
Nov 03 07:30:02.061271 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:30:02.061237    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7a5180_e91f_4bec_815a_a870c63c3992.slice/crio-a026f58ba68f1ed55ca20f54a876112f9fe120376a1044b6e5a1d333e020d170 WatchSource:0}: Error finding container a026f58ba68f1ed55ca20f54a876112f9fe120376a1044b6e5a1d333e020d170: Status 404 returned error can't find the container with id a026f58ba68f1ed55ca20f54a876112f9fe120376a1044b6e5a1d333e020d170
Nov 03 07:30:02.062518 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:30:02.062489    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e89e6f9_f776_4d44_9643_23c2d7e2922b.slice/crio-0ba49fb53f7b5fc99c56af2e4dd779f97fc88c671c5085e7673dc9b3e8fabe81 WatchSource:0}: Error finding container 0ba49fb53f7b5fc99c56af2e4dd779f97fc88c671c5085e7673dc9b3e8fabe81: Status 404 returned error can't find the container with id 0ba49fb53f7b5fc99c56af2e4dd779f97fc88c671c5085e7673dc9b3e8fabe81
Nov 03 07:30:02.063590 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:30:02.063557    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cb7b51_48dc_4368_98a3_640982b6e68e.slice/crio-cef8ca137ac9a552c5c0199dbabf9389c21e5e5c0618f730b7503bde22501aaa WatchSource:0}: Error finding container cef8ca137ac9a552c5c0199dbabf9389c21e5e5c0618f730b7503bde22501aaa: Status 404 returned error can't find the container with id cef8ca137ac9a552c5c0199dbabf9389c21e5e5c0618f730b7503bde22501aaa
Nov 03 07:30:02.116600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.116213    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" event={"ID":"de7a5180-e91f-4bec-815a-a870c63c3992","Type":"ContainerStarted","Data":"a026f58ba68f1ed55ca20f54a876112f9fe120376a1044b6e5a1d333e020d170"}
Nov 03 07:30:02.118433 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.118403    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs" event={"ID":"e505ebba-8ed9-4532-988d-5d3ff724b522","Type":"ContainerStarted","Data":"ada614b0f66291f5e481a47ecf932cc2a5fd5b5576da949e19d83b662e8a298b"}
Nov 03 07:30:02.118656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.118634    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs" event={"ID":"e505ebba-8ed9-4532-988d-5d3ff724b522","Type":"ContainerStarted","Data":"aae0a8e671a9ae2adef6e5d32c6b1f86262eca37c045517b82e68ed5d1eb2fb5"}
Nov 03 07:30:02.121011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.120975    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" event={"ID":"8e89e6f9-f776-4d44-9643-23c2d7e2922b","Type":"ContainerStarted","Data":"0ba49fb53f7b5fc99c56af2e4dd779f97fc88c671c5085e7673dc9b3e8fabe81"}
Nov 03 07:30:02.122529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.122481    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" event={"ID":"e3cb7b51-48dc-4368-98a3-640982b6e68e","Type":"ContainerStarted","Data":"cef8ca137ac9a552c5c0199dbabf9389c21e5e5c0618f730b7503bde22501aaa"}
Nov 03 07:30:02.124432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.124405    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" event={"ID":"d32f79ba-6fb3-447b-b224-639f835f6d69","Type":"ContainerStarted","Data":"f2eeda001f9140fd3b3a9447e955ebc51e7ae943a2f811805727605858b03ae4"}
Nov 03 07:30:02.126362 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.126326    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" event={"ID":"07659463-2d5f-4171-9e7b-d3b83cd4b838","Type":"ContainerStarted","Data":"985167dacf7b8075ef0b18950fdbb5bbbecb7cd0882a556bd97af4e391aaeeea"}
Nov 03 07:30:02.127916 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.127883    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" event={"ID":"bf996b97-74f9-450d-9ca9-826560308eb2","Type":"ContainerStarted","Data":"1fcd935ae83629c56895c3cd72dc05614e4634a4f2e7d82bbcb3d278f464dde1"}
Nov 03 07:30:02.129354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.129333    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf" event={"ID":"dae496e2-15ee-4ac2-8de0-b61d610f0150","Type":"ContainerStarted","Data":"800df364d6d1c0fc62fb275d92df8ec7b3bfcdc964a9048f56a1af4a1c036a0e"}
Nov 03 07:30:02.131051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.131027    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" event={"ID":"10559d66-8937-43fe-9f78-953042895f8a","Type":"ContainerStarted","Data":"a4be657c03675b69f477af8f169eb7afbfe73fc8ed23034c46b74159be951858"}
Nov 03 07:30:02.242071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:02.241963    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs" podStartSLOduration=2.241947385 podStartE2EDuration="2.241947385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:02.239088109 +0000 UTC m=+1854.571457151" watchObservedRunningTime="2026-11-03 07:30:02.241947385 +0000 UTC m=+1854.574316379"
Nov 03 07:30:03.234314 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.234274    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" event={"ID":"8e89e6f9-f776-4d44-9643-23c2d7e2922b","Type":"ContainerStarted","Data":"ab6b7418dbf704f8470da846e350bd0d24eb30b7bb2cb60c64bf434fa0341673"}
Nov 03 07:30:03.238099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.237840    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" event={"ID":"e3cb7b51-48dc-4368-98a3-640982b6e68e","Type":"ContainerStarted","Data":"2b0a5a75818910957b745dbada668422e139ac85f8ea0f08758badf3709cd55d"}
Nov 03 07:30:03.241578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.240590    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" event={"ID":"d32f79ba-6fb3-447b-b224-639f835f6d69","Type":"ContainerStarted","Data":"51174f9fe1174b87bf52f941bb94cc4323de448fa05c9d9452b4dc38a4b8b118"}
Nov 03 07:30:03.241578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.241055    4470 scope.go:122] "RemoveContainer" containerID="51174f9fe1174b87bf52f941bb94cc4323de448fa05c9d9452b4dc38a4b8b118"
Nov 03 07:30:03.252372 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.252340    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" event={"ID":"07659463-2d5f-4171-9e7b-d3b83cd4b838","Type":"ContainerStarted","Data":"21ece863ca0a6f064b8c6935b45728567a40fef325bcbaca853b98d34de12389"}
Nov 03 07:30:03.252815 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.252794    4470 scope.go:122] "RemoveContainer" containerID="21ece863ca0a6f064b8c6935b45728567a40fef325bcbaca853b98d34de12389"
Nov 03 07:30:03.274174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.270730    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" event={"ID":"bf996b97-74f9-450d-9ca9-826560308eb2","Type":"ContainerStarted","Data":"950b99922b474f2603544c527f740acc82cbaa7c3571814e848fac19dfdb117d"}
Nov 03 07:30:03.274174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.271180    4470 scope.go:122] "RemoveContainer" containerID="950b99922b474f2603544c527f740acc82cbaa7c3571814e848fac19dfdb117d"
Nov 03 07:30:03.275827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.275711    4470 generic.go:356] "Generic (PLEG): container finished" podID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerID="201edb9ab2bfd48f763a5cfc7d9ea40c614029ba8cb999e7617853b66602eadf" exitCode=60
Nov 03 07:30:03.275827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.275818    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf" event={"ID":"dae496e2-15ee-4ac2-8de0-b61d610f0150","Type":"ContainerDied","Data":"201edb9ab2bfd48f763a5cfc7d9ea40c614029ba8cb999e7617853b66602eadf"}
Nov 03 07:30:03.276610 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.276589    4470 scope.go:122] "RemoveContainer" containerID="201edb9ab2bfd48f763a5cfc7d9ea40c614029ba8cb999e7617853b66602eadf"
Nov 03 07:30:03.277564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.277542    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" event={"ID":"10559d66-8937-43fe-9f78-953042895f8a","Type":"ContainerStarted","Data":"d322b3f859c900d090eaa3de3afc6866dc53c020c3007d2e551fb9411a64d6a7"}
Nov 03 07:30:03.280168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.279953    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" event={"ID":"de7a5180-e91f-4bec-815a-a870c63c3992","Type":"ContainerStarted","Data":"7bd2a9a65f5e4ecf2c04b467ee38a9545c36e56088e56a345b8118c59c493c44"}
Nov 03 07:30:03.281916 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.281896    4470 generic.go:356] "Generic (PLEG): container finished" podID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerID="ada614b0f66291f5e481a47ecf932cc2a5fd5b5576da949e19d83b662e8a298b" exitCode=60
Nov 03 07:30:03.282005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.281956    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs" event={"ID":"e505ebba-8ed9-4532-988d-5d3ff724b522","Type":"ContainerDied","Data":"ada614b0f66291f5e481a47ecf932cc2a5fd5b5576da949e19d83b662e8a298b"}
Nov 03 07:30:03.282206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.282188    4470 scope.go:122] "RemoveContainer" containerID="ada614b0f66291f5e481a47ecf932cc2a5fd5b5576da949e19d83b662e8a298b"
Nov 03 07:30:03.355702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.355478    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" podStartSLOduration=3.355458063 podStartE2EDuration="3.355458063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:03.354864718 +0000 UTC m=+1855.687233711" watchObservedRunningTime="2026-11-03 07:30:03.355458063 +0000 UTC m=+1855.687827058"
Nov 03 07:30:03.407306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.407270    4470 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:30:03.407460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.407341    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:30:03.839827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.839760    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" podStartSLOduration=3.8397432179999997 podStartE2EDuration="3.839743218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:03.838776686 +0000 UTC m=+1856.171145701" watchObservedRunningTime="2026-11-03 07:30:03.839743218 +0000 UTC m=+1856.172112213"
Nov 03 07:30:03.964490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:03.964385    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" podStartSLOduration=3.964369095 podStartE2EDuration="3.964369095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:03.960220054 +0000 UTC m=+1856.292589050" watchObservedRunningTime="2026-11-03 07:30:03.964369095 +0000 UTC m=+1856.296738089"
Nov 03 07:30:04.292946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.292905    4470 generic.go:356] "Generic (PLEG): container finished" podID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerID="ab6b7418dbf704f8470da846e350bd0d24eb30b7bb2cb60c64bf434fa0341673" exitCode=7
Nov 03 07:30:04.293479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.293050    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" event={"ID":"8e89e6f9-f776-4d44-9643-23c2d7e2922b","Type":"ContainerDied","Data":"ab6b7418dbf704f8470da846e350bd0d24eb30b7bb2cb60c64bf434fa0341673"}
Nov 03 07:30:04.293479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.293386    4470 scope.go:122] "RemoveContainer" containerID="ab6b7418dbf704f8470da846e350bd0d24eb30b7bb2cb60c64bf434fa0341673"
Nov 03 07:30:04.295106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.295086    4470 generic.go:356] "Generic (PLEG): container finished" podID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerID="2b0a5a75818910957b745dbada668422e139ac85f8ea0f08758badf3709cd55d" exitCode=60
Nov 03 07:30:04.295247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.295127    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" event={"ID":"e3cb7b51-48dc-4368-98a3-640982b6e68e","Type":"ContainerDied","Data":"2b0a5a75818910957b745dbada668422e139ac85f8ea0f08758badf3709cd55d"}
Nov 03 07:30:04.295567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.295508    4470 scope.go:122] "RemoveContainer" containerID="2b0a5a75818910957b745dbada668422e139ac85f8ea0f08758badf3709cd55d"
Nov 03 07:30:04.299411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.298755    4470 generic.go:356] "Generic (PLEG): container finished" podID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerID="51174f9fe1174b87bf52f941bb94cc4323de448fa05c9d9452b4dc38a4b8b118" exitCode=60
Nov 03 07:30:04.299411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.298820    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" event={"ID":"d32f79ba-6fb3-447b-b224-639f835f6d69","Type":"ContainerDied","Data":"51174f9fe1174b87bf52f941bb94cc4323de448fa05c9d9452b4dc38a4b8b118"}
Nov 03 07:30:04.299411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.298855    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" event={"ID":"d32f79ba-6fb3-447b-b224-639f835f6d69","Type":"ContainerStarted","Data":"8d08192e62d65c3684e37c94c415730d7d33e472ff18995b5acf61ec7e6c745e"}
Nov 03 07:30:04.301782 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.301170    4470 generic.go:356] "Generic (PLEG): container finished" podID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerID="21ece863ca0a6f064b8c6935b45728567a40fef325bcbaca853b98d34de12389" exitCode=60
Nov 03 07:30:04.301782 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.301261    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" event={"ID":"07659463-2d5f-4171-9e7b-d3b83cd4b838","Type":"ContainerDied","Data":"21ece863ca0a6f064b8c6935b45728567a40fef325bcbaca853b98d34de12389"}
Nov 03 07:30:04.301782 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.301302    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" event={"ID":"07659463-2d5f-4171-9e7b-d3b83cd4b838","Type":"ContainerStarted","Data":"5a209d3f9a5aa5c563fafb92e7998ededf85ff7ca81fdc9e8ce561a96fcdff85"}
Nov 03 07:30:04.308097 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.308050    4470 generic.go:356] "Generic (PLEG): container finished" podID="bf996b97-74f9-450d-9ca9-826560308eb2" containerID="950b99922b474f2603544c527f740acc82cbaa7c3571814e848fac19dfdb117d" exitCode=60
Nov 03 07:30:04.308238 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.308117    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" event={"ID":"bf996b97-74f9-450d-9ca9-826560308eb2","Type":"ContainerDied","Data":"950b99922b474f2603544c527f740acc82cbaa7c3571814e848fac19dfdb117d"}
Nov 03 07:30:04.308238 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.308191    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" event={"ID":"bf996b97-74f9-450d-9ca9-826560308eb2","Type":"ContainerStarted","Data":"7fdd9c262f9f5ac52d5e7b8b6a38b64b1700598b74a4e730c5c075172453e0ae"}
Nov 03 07:30:04.310731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.310700    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf" event={"ID":"dae496e2-15ee-4ac2-8de0-b61d610f0150","Type":"ContainerStarted","Data":"517e975db0ab88131f7fb5d11005a6cc7a3b129c739d3c2bfaeb6254dd978713"}
Nov 03 07:30:04.312845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.312823    4470 generic.go:356] "Generic (PLEG): container finished" podID="10559d66-8937-43fe-9f78-953042895f8a" containerID="d322b3f859c900d090eaa3de3afc6866dc53c020c3007d2e551fb9411a64d6a7" exitCode=60
Nov 03 07:30:04.312943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.312901    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" event={"ID":"10559d66-8937-43fe-9f78-953042895f8a","Type":"ContainerDied","Data":"d322b3f859c900d090eaa3de3afc6866dc53c020c3007d2e551fb9411a64d6a7"}
Nov 03 07:30:04.313160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.313111    4470 scope.go:122] "RemoveContainer" containerID="d322b3f859c900d090eaa3de3afc6866dc53c020c3007d2e551fb9411a64d6a7"
Nov 03 07:30:04.314875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.314857    4470 generic.go:356] "Generic (PLEG): container finished" podID="de7a5180-e91f-4bec-815a-a870c63c3992" containerID="7bd2a9a65f5e4ecf2c04b467ee38a9545c36e56088e56a345b8118c59c493c44" exitCode=60
Nov 03 07:30:04.314968 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.314927    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" event={"ID":"de7a5180-e91f-4bec-815a-a870c63c3992","Type":"ContainerDied","Data":"7bd2a9a65f5e4ecf2c04b467ee38a9545c36e56088e56a345b8118c59c493c44"}
Nov 03 07:30:04.315243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.315229    4470 scope.go:122] "RemoveContainer" containerID="7bd2a9a65f5e4ecf2c04b467ee38a9545c36e56088e56a345b8118c59c493c44"
Nov 03 07:30:04.317641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.317622    4470 generic.go:356] "Generic (PLEG): container finished" podID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerID="b2837770a1c56f2fac77add84e5c66f07614f005fdd17b80d04afec1927474a6" exitCode=60
Nov 03 07:30:04.317733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.317695    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs" event={"ID":"e505ebba-8ed9-4532-988d-5d3ff724b522","Type":"ContainerDied","Data":"b2837770a1c56f2fac77add84e5c66f07614f005fdd17b80d04afec1927474a6"}
Nov 03 07:30:04.317777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.317740    4470 scope.go:122] "RemoveContainer" containerID="ada614b0f66291f5e481a47ecf932cc2a5fd5b5576da949e19d83b662e8a298b"
Nov 03 07:30:04.318052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.318034    4470 scope.go:122] "RemoveContainer" containerID="b2837770a1c56f2fac77add84e5c66f07614f005fdd17b80d04afec1927474a6"
Nov 03 07:30:04.318447 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:04.318422    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs_odh-ai-gateway-infra(e505ebba-8ed9-4532-988d-5d3ff724b522)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522"
Nov 03 07:30:04.368581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.368532    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" podStartSLOduration=4.36851686 podStartE2EDuration="4.36851686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:04.356639864 +0000 UTC m=+1856.689008860" watchObservedRunningTime="2026-11-03 07:30:04.36851686 +0000 UTC m=+1856.700885854"
Nov 03 07:30:04.482185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.479269    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" podStartSLOduration=4.479248603 podStartE2EDuration="4.479248603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:04.476098832 +0000 UTC m=+1856.808467826" watchObservedRunningTime="2026-11-03 07:30:04.479248603 +0000 UTC m=+1856.811617598"
Nov 03 07:30:04.771711 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:04.771639    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:30:04.978889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:04.978828    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" podStartSLOduration=4.97881218 podStartE2EDuration="4.97881218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:04.976600656 +0000 UTC m=+1857.308969649" watchObservedRunningTime="2026-11-03 07:30:04.97881218 +0000 UTC m=+1857.311181173"
Nov 03 07:30:05.100510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.100453    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf" podStartSLOduration=5.100438355 podStartE2EDuration="5.100438355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:05.096862683 +0000 UTC m=+1857.429231677" watchObservedRunningTime="2026-11-03 07:30:05.100438355 +0000 UTC m=+1857.432807362"
Nov 03 07:30:05.329332 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.329293    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" event={"ID":"de7a5180-e91f-4bec-815a-a870c63c3992","Type":"ContainerStarted","Data":"e8b455440407cdd68f593ff92d1cae1d224ac2cddfa6454c07e7255f115950b8"}
Nov 03 07:30:05.331403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.331382    4470 scope.go:122] "RemoveContainer" containerID="b2837770a1c56f2fac77add84e5c66f07614f005fdd17b80d04afec1927474a6"
Nov 03 07:30:05.331749 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:05.331726    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs_odh-ai-gateway-infra(e505ebba-8ed9-4532-988d-5d3ff724b522)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522"
Nov 03 07:30:05.333321 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.333297    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" event={"ID":"8e89e6f9-f776-4d44-9643-23c2d7e2922b","Type":"ContainerStarted","Data":"6b05b31ae213475c3d55fa97e45f15bd5fd9de4b6f9e38d445476e0af8cb0c87"}
Nov 03 07:30:05.335280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.335256    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" event={"ID":"e3cb7b51-48dc-4368-98a3-640982b6e68e","Type":"ContainerStarted","Data":"3d68134882edb78ce985695b5516af276bfc1dd45f8199cd69a4d8f74c2e0d87"}
Nov 03 07:30:05.338346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.338318    4470 generic.go:356] "Generic (PLEG): container finished" podID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerID="8d08192e62d65c3684e37c94c415730d7d33e472ff18995b5acf61ec7e6c745e" exitCode=60
Nov 03 07:30:05.338720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.338704    4470 scope.go:122] "RemoveContainer" containerID="8d08192e62d65c3684e37c94c415730d7d33e472ff18995b5acf61ec7e6c745e"
Nov 03 07:30:05.339032 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:05.339010    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6_odh-ai-gateway-infra(d32f79ba-6fb3-447b-b224-639f835f6d69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69"
Nov 03 07:30:05.339087 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.339071    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" event={"ID":"d32f79ba-6fb3-447b-b224-639f835f6d69","Type":"ContainerDied","Data":"8d08192e62d65c3684e37c94c415730d7d33e472ff18995b5acf61ec7e6c745e"}
Nov 03 07:30:05.339128 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.339114    4470 scope.go:122] "RemoveContainer" containerID="51174f9fe1174b87bf52f941bb94cc4323de448fa05c9d9452b4dc38a4b8b118"
Nov 03 07:30:05.341813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.341268    4470 generic.go:356] "Generic (PLEG): container finished" podID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerID="5a209d3f9a5aa5c563fafb92e7998ededf85ff7ca81fdc9e8ce561a96fcdff85" exitCode=60
Nov 03 07:30:05.341813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.341362    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" event={"ID":"07659463-2d5f-4171-9e7b-d3b83cd4b838","Type":"ContainerDied","Data":"5a209d3f9a5aa5c563fafb92e7998ededf85ff7ca81fdc9e8ce561a96fcdff85"}
Nov 03 07:30:05.341813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.341615    4470 scope.go:122] "RemoveContainer" containerID="5a209d3f9a5aa5c563fafb92e7998ededf85ff7ca81fdc9e8ce561a96fcdff85"
Nov 03 07:30:05.342079 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:05.341928    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2_odh-ai-gateway-infra(07659463-2d5f-4171-9e7b-d3b83cd4b838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838"
Nov 03 07:30:05.343743 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.343719    4470 generic.go:356] "Generic (PLEG): container finished" podID="bf996b97-74f9-450d-9ca9-826560308eb2" containerID="7fdd9c262f9f5ac52d5e7b8b6a38b64b1700598b74a4e730c5c075172453e0ae" exitCode=60
Nov 03 07:30:05.343743 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.343747    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" event={"ID":"bf996b97-74f9-450d-9ca9-826560308eb2","Type":"ContainerDied","Data":"7fdd9c262f9f5ac52d5e7b8b6a38b64b1700598b74a4e730c5c075172453e0ae"}
Nov 03 07:30:05.344070 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.344050    4470 scope.go:122] "RemoveContainer" containerID="7fdd9c262f9f5ac52d5e7b8b6a38b64b1700598b74a4e730c5c075172453e0ae"
Nov 03 07:30:05.344393 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:05.344372    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8_odh-ai-gateway-infra(bf996b97-74f9-450d-9ca9-826560308eb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" podUID="bf996b97-74f9-450d-9ca9-826560308eb2"
Nov 03 07:30:05.347044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.346716    4470 generic.go:356] "Generic (PLEG): container finished" podID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerID="517e975db0ab88131f7fb5d11005a6cc7a3b129c739d3c2bfaeb6254dd978713" exitCode=60
Nov 03 07:30:05.347044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.346808    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf" event={"ID":"dae496e2-15ee-4ac2-8de0-b61d610f0150","Type":"ContainerDied","Data":"517e975db0ab88131f7fb5d11005a6cc7a3b129c739d3c2bfaeb6254dd978713"}
Nov 03 07:30:05.347969 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.347362    4470 scope.go:122] "RemoveContainer" containerID="517e975db0ab88131f7fb5d11005a6cc7a3b129c739d3c2bfaeb6254dd978713"
Nov 03 07:30:05.347969 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:05.347638    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf_odh-ai-gateway-infra(dae496e2-15ee-4ac2-8de0-b61d610f0150)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150"
Nov 03 07:30:05.350422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.350026    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" event={"ID":"10559d66-8937-43fe-9f78-953042895f8a","Type":"ContainerStarted","Data":"0446e63fdc2e0cb7af0cb54b0756c37cf64644f1c3f9baa169eb5d79faeaee10"}
Nov 03 07:30:05.362763 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.362739    4470 scope.go:122] "RemoveContainer" containerID="21ece863ca0a6f064b8c6935b45728567a40fef325bcbaca853b98d34de12389"
Nov 03 07:30:05.378539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.378512    4470 scope.go:122] "RemoveContainer" containerID="950b99922b474f2603544c527f740acc82cbaa7c3571814e848fac19dfdb117d"
Nov 03 07:30:05.393452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.393368    4470 scope.go:122] "RemoveContainer" containerID="201edb9ab2bfd48f763a5cfc7d9ea40c614029ba8cb999e7617853b66602eadf"
Nov 03 07:30:05.462992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:05.462936    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" podStartSLOduration=5.462921114 podStartE2EDuration="5.462921114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:05.461178492 +0000 UTC m=+1857.793547480" watchObservedRunningTime="2026-11-03 07:30:05.462921114 +0000 UTC m=+1857.795290107"
Nov 03 07:30:06.361853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.361819    4470 generic.go:356] "Generic (PLEG): container finished" podID="de7a5180-e91f-4bec-815a-a870c63c3992" containerID="e8b455440407cdd68f593ff92d1cae1d224ac2cddfa6454c07e7255f115950b8" exitCode=60
Nov 03 07:30:06.362364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.361905    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" event={"ID":"de7a5180-e91f-4bec-815a-a870c63c3992","Type":"ContainerDied","Data":"e8b455440407cdd68f593ff92d1cae1d224ac2cddfa6454c07e7255f115950b8"}
Nov 03 07:30:06.362364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.361964    4470 scope.go:122] "RemoveContainer" containerID="7bd2a9a65f5e4ecf2c04b467ee38a9545c36e56088e56a345b8118c59c493c44"
Nov 03 07:30:06.362364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.362290    4470 scope.go:122] "RemoveContainer" containerID="e8b455440407cdd68f593ff92d1cae1d224ac2cddfa6454c07e7255f115950b8"
Nov 03 07:30:06.362743 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:06.362714    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894850-56g2c_odh-ai-gateway-infra(de7a5180-e91f-4bec-815a-a870c63c3992)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" podUID="de7a5180-e91f-4bec-815a-a870c63c3992"
Nov 03 07:30:06.364344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.364320    4470 generic.go:356] "Generic (PLEG): container finished" podID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerID="6b05b31ae213475c3d55fa97e45f15bd5fd9de4b6f9e38d445476e0af8cb0c87" exitCode=60
Nov 03 07:30:06.364435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.364414    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" event={"ID":"8e89e6f9-f776-4d44-9643-23c2d7e2922b","Type":"ContainerDied","Data":"6b05b31ae213475c3d55fa97e45f15bd5fd9de4b6f9e38d445476e0af8cb0c87"}
Nov 03 07:30:06.364685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.364668    4470 scope.go:122] "RemoveContainer" containerID="6b05b31ae213475c3d55fa97e45f15bd5fd9de4b6f9e38d445476e0af8cb0c87"
Nov 03 07:30:06.365046 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:06.365026    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb_odh-ai-gateway-infra(8e89e6f9-f776-4d44-9643-23c2d7e2922b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b"
Nov 03 07:30:06.366448 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.366426    4470 generic.go:356] "Generic (PLEG): container finished" podID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerID="3d68134882edb78ce985695b5516af276bfc1dd45f8199cd69a4d8f74c2e0d87" exitCode=60
Nov 03 07:30:06.366559 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.366535    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" event={"ID":"e3cb7b51-48dc-4368-98a3-640982b6e68e","Type":"ContainerDied","Data":"3d68134882edb78ce985695b5516af276bfc1dd45f8199cd69a4d8f74c2e0d87"}
Nov 03 07:30:06.366795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.366777    4470 scope.go:122] "RemoveContainer" containerID="3d68134882edb78ce985695b5516af276bfc1dd45f8199cd69a4d8f74c2e0d87"
Nov 03 07:30:06.367116 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:06.367093    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw_odh-ai-gateway-infra(e3cb7b51-48dc-4368-98a3-640982b6e68e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e"
Nov 03 07:30:06.368639 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.368618    4470 scope.go:122] "RemoveContainer" containerID="8d08192e62d65c3684e37c94c415730d7d33e472ff18995b5acf61ec7e6c745e"
Nov 03 07:30:06.368946 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:06.368914    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6_odh-ai-gateway-infra(d32f79ba-6fb3-447b-b224-639f835f6d69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69"
Nov 03 07:30:06.370501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.370458    4470 scope.go:122] "RemoveContainer" containerID="5a209d3f9a5aa5c563fafb92e7998ededf85ff7ca81fdc9e8ce561a96fcdff85"
Nov 03 07:30:06.370779 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:06.370759    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2_odh-ai-gateway-infra(07659463-2d5f-4171-9e7b-d3b83cd4b838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838"
Nov 03 07:30:06.372304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.372283    4470 scope.go:122] "RemoveContainer" containerID="7fdd9c262f9f5ac52d5e7b8b6a38b64b1700598b74a4e730c5c075172453e0ae"
Nov 03 07:30:06.372552 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:06.372528    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8_odh-ai-gateway-infra(bf996b97-74f9-450d-9ca9-826560308eb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" podUID="bf996b97-74f9-450d-9ca9-826560308eb2"
Nov 03 07:30:06.374309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.374291    4470 scope.go:122] "RemoveContainer" containerID="517e975db0ab88131f7fb5d11005a6cc7a3b129c739d3c2bfaeb6254dd978713"
Nov 03 07:30:06.374607 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:06.374582    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf_odh-ai-gateway-infra(dae496e2-15ee-4ac2-8de0-b61d610f0150)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150"
Nov 03 07:30:06.376005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.375985    4470 generic.go:356] "Generic (PLEG): container finished" podID="10559d66-8937-43fe-9f78-953042895f8a" containerID="0446e63fdc2e0cb7af0cb54b0756c37cf64644f1c3f9baa169eb5d79faeaee10" exitCode=60
Nov 03 07:30:06.376076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.376036    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" event={"ID":"10559d66-8937-43fe-9f78-953042895f8a","Type":"ContainerDied","Data":"0446e63fdc2e0cb7af0cb54b0756c37cf64644f1c3f9baa169eb5d79faeaee10"}
Nov 03 07:30:06.376373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.376356    4470 scope.go:122] "RemoveContainer" containerID="0446e63fdc2e0cb7af0cb54b0756c37cf64644f1c3f9baa169eb5d79faeaee10"
Nov 03 07:30:06.376699 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:06.376676    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2_odh-ai-gateway-infra(10559d66-8937-43fe-9f78-953042895f8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" podUID="10559d66-8937-43fe-9f78-953042895f8a"
Nov 03 07:30:06.378052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.378035    4470 scope.go:122] "RemoveContainer" containerID="ab6b7418dbf704f8470da846e350bd0d24eb30b7bb2cb60c64bf434fa0341673"
Nov 03 07:30:06.392423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.392399    4470 scope.go:122] "RemoveContainer" containerID="2b0a5a75818910957b745dbada668422e139ac85f8ea0f08758badf3709cd55d"
Nov 03 07:30:06.406765 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:06.406741    4470 scope.go:122] "RemoveContainer" containerID="d322b3f859c900d090eaa3de3afc6866dc53c020c3007d2e551fb9411a64d6a7"
Nov 03 07:30:07.386646 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:07.386565    4470 scope.go:122] "RemoveContainer" containerID="e8b455440407cdd68f593ff92d1cae1d224ac2cddfa6454c07e7255f115950b8"
Nov 03 07:30:07.387180 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:07.387007    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894850-56g2c_odh-ai-gateway-infra(de7a5180-e91f-4bec-815a-a870c63c3992)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" podUID="de7a5180-e91f-4bec-815a-a870c63c3992"
Nov 03 07:30:07.388525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:07.388501    4470 scope.go:122] "RemoveContainer" containerID="6b05b31ae213475c3d55fa97e45f15bd5fd9de4b6f9e38d445476e0af8cb0c87"
Nov 03 07:30:07.388919 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:07.388893    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb_odh-ai-gateway-infra(8e89e6f9-f776-4d44-9643-23c2d7e2922b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b"
Nov 03 07:30:07.390284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:07.390247    4470 scope.go:122] "RemoveContainer" containerID="3d68134882edb78ce985695b5516af276bfc1dd45f8199cd69a4d8f74c2e0d87"
Nov 03 07:30:07.390608 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:07.390580    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw_odh-ai-gateway-infra(e3cb7b51-48dc-4368-98a3-640982b6e68e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e"
Nov 03 07:30:07.391637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:07.391617    4470 generic.go:356] "Generic (PLEG): container finished" podID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerID="35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d" exitCode=0
Nov 03 07:30:07.391743 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:07.391680    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" event={"ID":"eabd31f7-12a2-4753-8a37-bc835e84615d","Type":"ContainerDied","Data":"35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d"}
Nov 03 07:30:07.392152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:07.392119    4470 scope.go:122] "RemoveContainer" containerID="35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d"
Nov 03 07:30:07.393846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:07.393819    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:30:07.393926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:07.393885    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:30:07.393967 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:07.393925    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:30:07.393967 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:07.393936    4470 scope.go:122] "RemoveContainer" containerID="0446e63fdc2e0cb7af0cb54b0756c37cf64644f1c3f9baa169eb5d79faeaee10"
Nov 03 07:30:07.394273 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:07.394248    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2_odh-ai-gateway-infra(10559d66-8937-43fe-9f78-953042895f8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" podUID="10559d66-8937-43fe-9f78-953042895f8a"
Nov 03 07:30:08.404520 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:08.404486    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" event={"ID":"eabd31f7-12a2-4753-8a37-bc835e84615d","Type":"ContainerStarted","Data":"7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36"}
Nov 03 07:30:08.542808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:08.542772    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:30:08.542998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:08.542829    4470 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 07:30:08.542998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:08.542878    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 07:30:09.544002 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:09.543965    4470 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 07:30:09.544452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:09.544029    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 07:30:10.542587 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:10.542556    4470 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 07:30:10.542794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:10.542607    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 07:30:11.543252 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:11.543215    4470 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 07:30:11.543742 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:11.543277    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 07:30:12.542820 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:12.542783    4470 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 07:30:12.542994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:12.542831    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 07:30:13.542733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:13.542694    4470 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 07:30:13.543154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:13.542752    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 07:30:14.547901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:14.547872    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:30:14.548378 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:14.548315    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:30:14.548868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:14.548852    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:30:14.800342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:14.800260    4470 scope.go:122] "RemoveContainer" containerID="b2837770a1c56f2fac77add84e5c66f07614f005fdd17b80d04afec1927474a6"
Nov 03 07:30:15.542123 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:15.542028    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs" event={"ID":"e505ebba-8ed9-4532-988d-5d3ff724b522","Type":"ContainerStarted","Data":"29b30ee441d2a48203c0ec4585b157d1e1870a448ed6fa59a10c339718a2e879"}
Nov 03 07:30:15.799485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:15.799402    4470 scope.go:122] "RemoveContainer" containerID="8d08192e62d65c3684e37c94c415730d7d33e472ff18995b5acf61ec7e6c745e"
Nov 03 07:30:15.799921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:15.799605    4470 scope.go:122] "RemoveContainer" containerID="517e975db0ab88131f7fb5d11005a6cc7a3b129c739d3c2bfaeb6254dd978713"
Nov 03 07:30:16.552270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:16.552189    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf" event={"ID":"dae496e2-15ee-4ac2-8de0-b61d610f0150","Type":"ContainerStarted","Data":"ddcb0f4902f48857fefe1323942b1c7119b1fd5ad266a696f412f1bb02eddd08"}
Nov 03 07:30:16.554541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:16.554509    4470 generic.go:356] "Generic (PLEG): container finished" podID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerID="29b30ee441d2a48203c0ec4585b157d1e1870a448ed6fa59a10c339718a2e879" exitCode=60
Nov 03 07:30:16.554697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:16.554580    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs" event={"ID":"e505ebba-8ed9-4532-988d-5d3ff724b522","Type":"ContainerDied","Data":"29b30ee441d2a48203c0ec4585b157d1e1870a448ed6fa59a10c339718a2e879"}
Nov 03 07:30:16.554697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:16.554648    4470 scope.go:122] "RemoveContainer" containerID="b2837770a1c56f2fac77add84e5c66f07614f005fdd17b80d04afec1927474a6"
Nov 03 07:30:16.555027 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:16.555011    4470 scope.go:122] "RemoveContainer" containerID="29b30ee441d2a48203c0ec4585b157d1e1870a448ed6fa59a10c339718a2e879"
Nov 03 07:30:16.555390 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:16.555368    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs_odh-ai-gateway-infra(e505ebba-8ed9-4532-988d-5d3ff724b522)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522"
Nov 03 07:30:16.556772 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:16.556753    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" event={"ID":"d32f79ba-6fb3-447b-b224-639f835f6d69","Type":"ContainerStarted","Data":"c5c80b58050d012454bd01e9f00799dc0e029722cde21716c7facdd9dcb7c911"}
Nov 03 07:30:16.683886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:16.683851    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"]
Nov 03 07:30:16.800188 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:16.799802    4470 scope.go:122] "RemoveContainer" containerID="5a209d3f9a5aa5c563fafb92e7998ededf85ff7ca81fdc9e8ce561a96fcdff85"
Nov 03 07:30:16.800678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:16.800564    4470 scope.go:122] "RemoveContainer" containerID="6b05b31ae213475c3d55fa97e45f15bd5fd9de4b6f9e38d445476e0af8cb0c87"
Nov 03 07:30:17.568178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.568120    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" event={"ID":"8e89e6f9-f776-4d44-9643-23c2d7e2922b","Type":"ContainerStarted","Data":"0bf16c2ba46a709869f5f1cb619b42b620e6826bb7373b74822ed03c5762735b"}
Nov 03 07:30:17.570245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.570210    4470 generic.go:356] "Generic (PLEG): container finished" podID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerID="c5c80b58050d012454bd01e9f00799dc0e029722cde21716c7facdd9dcb7c911" exitCode=60
Nov 03 07:30:17.570390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.570252    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" event={"ID":"d32f79ba-6fb3-447b-b224-639f835f6d69","Type":"ContainerDied","Data":"c5c80b58050d012454bd01e9f00799dc0e029722cde21716c7facdd9dcb7c911"}
Nov 03 07:30:17.570390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.570306    4470 scope.go:122] "RemoveContainer" containerID="8d08192e62d65c3684e37c94c415730d7d33e472ff18995b5acf61ec7e6c745e"
Nov 03 07:30:17.570606 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.570588    4470 scope.go:122] "RemoveContainer" containerID="c5c80b58050d012454bd01e9f00799dc0e029722cde21716c7facdd9dcb7c911"
Nov 03 07:30:17.570901 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:17.570878    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6_odh-ai-gateway-infra(d32f79ba-6fb3-447b-b224-639f835f6d69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69"
Nov 03 07:30:17.572327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.572305    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" event={"ID":"07659463-2d5f-4171-9e7b-d3b83cd4b838","Type":"ContainerStarted","Data":"7adad0ba287f048d32e0068f7d26756879f6a6bee3def5d28795c648523b71f3"}
Nov 03 07:30:17.574222 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.574199    4470 generic.go:356] "Generic (PLEG): container finished" podID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerID="ddcb0f4902f48857fefe1323942b1c7119b1fd5ad266a696f412f1bb02eddd08" exitCode=60
Nov 03 07:30:17.574293 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.574263    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf" event={"ID":"dae496e2-15ee-4ac2-8de0-b61d610f0150","Type":"ContainerDied","Data":"ddcb0f4902f48857fefe1323942b1c7119b1fd5ad266a696f412f1bb02eddd08"}
Nov 03 07:30:17.574576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.574558    4470 scope.go:122] "RemoveContainer" containerID="ddcb0f4902f48857fefe1323942b1c7119b1fd5ad266a696f412f1bb02eddd08"
Nov 03 07:30:17.574905 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:17.574883    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf_odh-ai-gateway-infra(dae496e2-15ee-4ac2-8de0-b61d610f0150)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150"
Nov 03 07:30:17.590702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.590468    4470 scope.go:122] "RemoveContainer" containerID="517e975db0ab88131f7fb5d11005a6cc7a3b129c739d3c2bfaeb6254dd978713"
Nov 03 07:30:17.699154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.696446    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"]
Nov 03 07:30:17.748608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.748581    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"
Nov 03 07:30:17.800280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.800254    4470 scope.go:122] "RemoveContainer" containerID="3d68134882edb78ce985695b5516af276bfc1dd45f8199cd69a4d8f74c2e0d87"
Nov 03 07:30:17.871025 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.870990    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e505ebba-8ed9-4532-988d-5d3ff724b522-kube-api-access-f4sbl\" (UniqueName: \"kubernetes.io/projected/e505ebba-8ed9-4532-988d-5d3ff724b522-kube-api-access-f4sbl\") pod \"e505ebba-8ed9-4532-988d-5d3ff724b522\" (UID: \"e505ebba-8ed9-4532-988d-5d3ff724b522\") "
Nov 03 07:30:17.871234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.871214    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e505ebba-8ed9-4532-988d-5d3ff724b522-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e505ebba-8ed9-4532-988d-5d3ff724b522-maas-api-service-ca\") pod \"e505ebba-8ed9-4532-988d-5d3ff724b522\" (UID: \"e505ebba-8ed9-4532-988d-5d3ff724b522\") "
Nov 03 07:30:17.871562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.871539    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e505ebba-8ed9-4532-988d-5d3ff724b522-maas-api-service-ca" pod "e505ebba-8ed9-4532-988d-5d3ff724b522" (UID: "e505ebba-8ed9-4532-988d-5d3ff724b522"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:30:17.872114 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.872094    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e505ebba-8ed9-4532-988d-5d3ff724b522-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:17.948259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.948086    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e505ebba-8ed9-4532-988d-5d3ff724b522-kube-api-access-f4sbl" pod "e505ebba-8ed9-4532-988d-5d3ff724b522" (UID: "e505ebba-8ed9-4532-988d-5d3ff724b522"). InnerVolumeSpecName "kube-api-access-f4sbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:30:17.952670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.952637    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"]
Nov 03 07:30:17.974420 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:17.974390    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4sbl\" (UniqueName: \"kubernetes.io/projected/e505ebba-8ed9-4532-988d-5d3ff724b522-kube-api-access-f4sbl\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:18.587570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.587538    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"
Nov 03 07:30:18.587752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.587721    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs" event={"ID":"e505ebba-8ed9-4532-988d-5d3ff724b522","Type":"ContainerDied","Data":"aae0a8e671a9ae2adef6e5d32c6b1f86262eca37c045517b82e68ed5d1eb2fb5"}
Nov 03 07:30:18.587869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.587786    4470 scope.go:122] "RemoveContainer" containerID="29b30ee441d2a48203c0ec4585b157d1e1870a448ed6fa59a10c339718a2e879"
Nov 03 07:30:18.590179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.590150    4470 generic.go:356] "Generic (PLEG): container finished" podID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerID="0bf16c2ba46a709869f5f1cb619b42b620e6826bb7373b74822ed03c5762735b" exitCode=60
Nov 03 07:30:18.590358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.590192    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" event={"ID":"8e89e6f9-f776-4d44-9643-23c2d7e2922b","Type":"ContainerDied","Data":"0bf16c2ba46a709869f5f1cb619b42b620e6826bb7373b74822ed03c5762735b"}
Nov 03 07:30:18.590682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.590661    4470 scope.go:122] "RemoveContainer" containerID="0bf16c2ba46a709869f5f1cb619b42b620e6826bb7373b74822ed03c5762735b"
Nov 03 07:30:18.591001 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:18.590978    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb_odh-ai-gateway-infra(8e89e6f9-f776-4d44-9643-23c2d7e2922b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b"
Nov 03 07:30:18.592840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.592819    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" event={"ID":"e3cb7b51-48dc-4368-98a3-640982b6e68e","Type":"ContainerStarted","Data":"d9a3260382e434f05a4b35eec670df1cd855a55a1994ed57f9bf5912e97f0a4e"}
Nov 03 07:30:18.600033 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.599995    4470 generic.go:356] "Generic (PLEG): container finished" podID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerID="7adad0ba287f048d32e0068f7d26756879f6a6bee3def5d28795c648523b71f3" exitCode=60
Nov 03 07:30:18.600172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.600065    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" event={"ID":"07659463-2d5f-4171-9e7b-d3b83cd4b838","Type":"ContainerDied","Data":"7adad0ba287f048d32e0068f7d26756879f6a6bee3def5d28795c648523b71f3"}
Nov 03 07:30:18.600495 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.600478    4470 scope.go:122] "RemoveContainer" containerID="7adad0ba287f048d32e0068f7d26756879f6a6bee3def5d28795c648523b71f3"
Nov 03 07:30:18.600836 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:18.600817    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2_odh-ai-gateway-infra(07659463-2d5f-4171-9e7b-d3b83cd4b838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838"
Nov 03 07:30:18.602604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.602583    4470 scope.go:122] "RemoveContainer" containerID="6b05b31ae213475c3d55fa97e45f15bd5fd9de4b6f9e38d445476e0af8cb0c87"
Nov 03 07:30:18.617879 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.617859    4470 scope.go:122] "RemoveContainer" containerID="5a209d3f9a5aa5c563fafb92e7998ededf85ff7ca81fdc9e8ce561a96fcdff85"
Nov 03 07:30:18.703810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.703782    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"]
Nov 03 07:30:18.800193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.800127    4470 scope.go:122] "RemoveContainer" containerID="7fdd9c262f9f5ac52d5e7b8b6a38b64b1700598b74a4e730c5c075172453e0ae"
Nov 03 07:30:18.800553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.800532    4470 scope.go:122] "RemoveContainer" containerID="0446e63fdc2e0cb7af0cb54b0756c37cf64644f1c3f9baa169eb5d79faeaee10"
Nov 03 07:30:18.827835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.827797    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"]
Nov 03 07:30:18.894581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.894557    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"
Nov 03 07:30:18.896814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.896794    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"
Nov 03 07:30:18.959273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.959240    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"]
Nov 03 07:30:18.970000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.969961    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-295fba7c-29894850-k5xgs"]
Nov 03 07:30:18.989299 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.989270    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32f79ba-6fb3-447b-b224-639f835f6d69-kube-api-access-s8tl7\" (UniqueName: \"kubernetes.io/projected/d32f79ba-6fb3-447b-b224-639f835f6d69-kube-api-access-s8tl7\") pod \"d32f79ba-6fb3-447b-b224-639f835f6d69\" (UID: \"d32f79ba-6fb3-447b-b224-639f835f6d69\") "
Nov 03 07:30:18.989497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.989390    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae496e2-15ee-4ac2-8de0-b61d610f0150-kube-api-access-wcw5m\" (UniqueName: \"kubernetes.io/projected/dae496e2-15ee-4ac2-8de0-b61d610f0150-kube-api-access-wcw5m\") pod \"dae496e2-15ee-4ac2-8de0-b61d610f0150\" (UID: \"dae496e2-15ee-4ac2-8de0-b61d610f0150\") "
Nov 03 07:30:18.989558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.989541    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32f79ba-6fb3-447b-b224-639f835f6d69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32f79ba-6fb3-447b-b224-639f835f6d69-maas-api-service-ca\") pod \"d32f79ba-6fb3-447b-b224-639f835f6d69\" (UID: \"d32f79ba-6fb3-447b-b224-639f835f6d69\") "
Nov 03 07:30:18.989609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.989579    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae496e2-15ee-4ac2-8de0-b61d610f0150-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae496e2-15ee-4ac2-8de0-b61d610f0150-maas-api-service-ca\") pod \"dae496e2-15ee-4ac2-8de0-b61d610f0150\" (UID: \"dae496e2-15ee-4ac2-8de0-b61d610f0150\") "
Nov 03 07:30:18.989974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.989942    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32f79ba-6fb3-447b-b224-639f835f6d69-maas-api-service-ca" pod "d32f79ba-6fb3-447b-b224-639f835f6d69" (UID: "d32f79ba-6fb3-447b-b224-639f835f6d69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:30:18.989974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.989962    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae496e2-15ee-4ac2-8de0-b61d610f0150-maas-api-service-ca" pod "dae496e2-15ee-4ac2-8de0-b61d610f0150" (UID: "dae496e2-15ee-4ac2-8de0-b61d610f0150"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:30:18.990237 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.990222    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32f79ba-6fb3-447b-b224-639f835f6d69-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:18.990287 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.990240    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae496e2-15ee-4ac2-8de0-b61d610f0150-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:18.991807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.991780    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae496e2-15ee-4ac2-8de0-b61d610f0150-kube-api-access-wcw5m" pod "dae496e2-15ee-4ac2-8de0-b61d610f0150" (UID: "dae496e2-15ee-4ac2-8de0-b61d610f0150"). InnerVolumeSpecName "kube-api-access-wcw5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:30:18.991973 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:18.991954    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32f79ba-6fb3-447b-b224-639f835f6d69-kube-api-access-s8tl7" pod "d32f79ba-6fb3-447b-b224-639f835f6d69" (UID: "d32f79ba-6fb3-447b-b224-639f835f6d69"). InnerVolumeSpecName "kube-api-access-s8tl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:30:19.092192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.092080    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcw5m\" (UniqueName: \"kubernetes.io/projected/dae496e2-15ee-4ac2-8de0-b61d610f0150-kube-api-access-wcw5m\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:19.092192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.092111    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8tl7\" (UniqueName: \"kubernetes.io/projected/d32f79ba-6fb3-447b-b224-639f835f6d69-kube-api-access-s8tl7\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:19.608964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.608923    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6" event={"ID":"d32f79ba-6fb3-447b-b224-639f835f6d69","Type":"ContainerDied","Data":"f2eeda001f9140fd3b3a9447e955ebc51e7ae943a2f811805727605858b03ae4"}
Nov 03 07:30:19.609197 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.609022    4470 scope.go:122] "RemoveContainer" containerID="c5c80b58050d012454bd01e9f00799dc0e029722cde21716c7facdd9dcb7c911"
Nov 03 07:30:19.609197 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.608939    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"
Nov 03 07:30:19.613205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.613168    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" event={"ID":"bf996b97-74f9-450d-9ca9-826560308eb2","Type":"ContainerStarted","Data":"d59a2c6470550efbd2406309c0124f8a4803a5c89e7b299aff594a5b0053fec7"}
Nov 03 07:30:19.614915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.614896    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"
Nov 03 07:30:19.615026 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.614904    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf" event={"ID":"dae496e2-15ee-4ac2-8de0-b61d610f0150","Type":"ContainerDied","Data":"800df364d6d1c0fc62fb275d92df8ec7b3bfcdc964a9048f56a1af4a1c036a0e"}
Nov 03 07:30:19.621839 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.620507    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" event={"ID":"10559d66-8937-43fe-9f78-953042895f8a","Type":"ContainerStarted","Data":"5268acc0bf7c1d3f952416798459d99a8f1c84ac5abf314f33e1176ffd547cfa"}
Nov 03 07:30:19.625034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.625015    4470 scope.go:122] "RemoveContainer" containerID="ddcb0f4902f48857fefe1323942b1c7119b1fd5ad266a696f412f1bb02eddd08"
Nov 03 07:30:19.628343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.628322    4470 generic.go:356] "Generic (PLEG): container finished" podID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerID="d9a3260382e434f05a4b35eec670df1cd855a55a1994ed57f9bf5912e97f0a4e" exitCode=60
Nov 03 07:30:19.628436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.628359    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" event={"ID":"e3cb7b51-48dc-4368-98a3-640982b6e68e","Type":"ContainerDied","Data":"d9a3260382e434f05a4b35eec670df1cd855a55a1994ed57f9bf5912e97f0a4e"}
Nov 03 07:30:19.628766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.628751    4470 scope.go:122] "RemoveContainer" containerID="d9a3260382e434f05a4b35eec670df1cd855a55a1994ed57f9bf5912e97f0a4e"
Nov 03 07:30:19.629066 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:19.629046    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw_odh-ai-gateway-infra(e3cb7b51-48dc-4368-98a3-640982b6e68e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e"
Nov 03 07:30:19.637774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.637753    4470 scope.go:122] "RemoveContainer" containerID="3d68134882edb78ce985695b5516af276bfc1dd45f8199cd69a4d8f74c2e0d87"
Nov 03 07:30:19.800483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.799598    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"
Nov 03 07:30:19.813452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.813426    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522" path="/var/lib/kubelet/pods/e505ebba-8ed9-4532-988d-5d3ff724b522/volumes"
Nov 03 07:30:19.828390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.828359    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"
Nov 03 07:30:19.904597 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.904505    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07659463-2d5f-4171-9e7b-d3b83cd4b838-kube-api-access-h562r\" (UniqueName: \"kubernetes.io/projected/07659463-2d5f-4171-9e7b-d3b83cd4b838-kube-api-access-h562r\") pod \"07659463-2d5f-4171-9e7b-d3b83cd4b838\" (UID: \"07659463-2d5f-4171-9e7b-d3b83cd4b838\") "
Nov 03 07:30:19.904759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.904679    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07659463-2d5f-4171-9e7b-d3b83cd4b838-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07659463-2d5f-4171-9e7b-d3b83cd4b838-maas-api-service-ca\") pod \"07659463-2d5f-4171-9e7b-d3b83cd4b838\" (UID: \"07659463-2d5f-4171-9e7b-d3b83cd4b838\") "
Nov 03 07:30:19.904759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.904717    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e89e6f9-f776-4d44-9643-23c2d7e2922b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e89e6f9-f776-4d44-9643-23c2d7e2922b-maas-api-service-ca\") pod \"8e89e6f9-f776-4d44-9643-23c2d7e2922b\" (UID: \"8e89e6f9-f776-4d44-9643-23c2d7e2922b\") "
Nov 03 07:30:19.904759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.904746    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e89e6f9-f776-4d44-9643-23c2d7e2922b-kube-api-access-crsnp\" (UniqueName: \"kubernetes.io/projected/8e89e6f9-f776-4d44-9643-23c2d7e2922b-kube-api-access-crsnp\") pod \"8e89e6f9-f776-4d44-9643-23c2d7e2922b\" (UID: \"8e89e6f9-f776-4d44-9643-23c2d7e2922b\") "
Nov 03 07:30:19.905067 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.905038    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07659463-2d5f-4171-9e7b-d3b83cd4b838-maas-api-service-ca" pod "07659463-2d5f-4171-9e7b-d3b83cd4b838" (UID: "07659463-2d5f-4171-9e7b-d3b83cd4b838"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:30:19.905167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.905080    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e89e6f9-f776-4d44-9643-23c2d7e2922b-maas-api-service-ca" pod "8e89e6f9-f776-4d44-9643-23c2d7e2922b" (UID: "8e89e6f9-f776-4d44-9643-23c2d7e2922b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:30:19.906578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.905911    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07659463-2d5f-4171-9e7b-d3b83cd4b838-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:19.906578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.905938    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e89e6f9-f776-4d44-9643-23c2d7e2922b-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:19.907059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.907037    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07659463-2d5f-4171-9e7b-d3b83cd4b838-kube-api-access-h562r" pod "07659463-2d5f-4171-9e7b-d3b83cd4b838" (UID: "07659463-2d5f-4171-9e7b-d3b83cd4b838"). InnerVolumeSpecName "kube-api-access-h562r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:30:19.907185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:19.907125    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e89e6f9-f776-4d44-9643-23c2d7e2922b-kube-api-access-crsnp" pod "8e89e6f9-f776-4d44-9643-23c2d7e2922b" (UID: "8e89e6f9-f776-4d44-9643-23c2d7e2922b"). InnerVolumeSpecName "kube-api-access-crsnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:30:20.007603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.007561    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crsnp\" (UniqueName: \"kubernetes.io/projected/8e89e6f9-f776-4d44-9643-23c2d7e2922b-kube-api-access-crsnp\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:20.007603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.007587    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h562r\" (UniqueName: \"kubernetes.io/projected/07659463-2d5f-4171-9e7b-d3b83cd4b838-kube-api-access-h562r\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:20.214073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.213992    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"]
Nov 03 07:30:20.299483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.299443    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"]
Nov 03 07:30:20.303764 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.303733    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-acdd08-29894850-kx4cf"]
Nov 03 07:30:20.536300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.536203    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"]
Nov 03 07:30:20.540475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.540448    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5183ae-29894850-9mwd6"]
Nov 03 07:30:20.638091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.638062    4470 generic.go:356] "Generic (PLEG): container finished" podID="10559d66-8937-43fe-9f78-953042895f8a" containerID="5268acc0bf7c1d3f952416798459d99a8f1c84ac5abf314f33e1176ffd547cfa" exitCode=60
Nov 03 07:30:20.638309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.638152    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" event={"ID":"10559d66-8937-43fe-9f78-953042895f8a","Type":"ContainerDied","Data":"5268acc0bf7c1d3f952416798459d99a8f1c84ac5abf314f33e1176ffd547cfa"}
Nov 03 07:30:20.638309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.638224    4470 scope.go:122] "RemoveContainer" containerID="0446e63fdc2e0cb7af0cb54b0756c37cf64644f1c3f9baa169eb5d79faeaee10"
Nov 03 07:30:20.638670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.638649    4470 scope.go:122] "RemoveContainer" containerID="5268acc0bf7c1d3f952416798459d99a8f1c84ac5abf314f33e1176ffd547cfa"
Nov 03 07:30:20.639040 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:20.639016    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2_odh-ai-gateway-infra(10559d66-8937-43fe-9f78-953042895f8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" podUID="10559d66-8937-43fe-9f78-953042895f8a"
Nov 03 07:30:20.639880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.639864    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"
Nov 03 07:30:20.639971 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.639879    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb" event={"ID":"8e89e6f9-f776-4d44-9643-23c2d7e2922b","Type":"ContainerDied","Data":"0ba49fb53f7b5fc99c56af2e4dd779f97fc88c671c5085e7673dc9b3e8fabe81"}
Nov 03 07:30:20.644485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.644464    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"
Nov 03 07:30:20.644587 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.644469    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2" event={"ID":"07659463-2d5f-4171-9e7b-d3b83cd4b838","Type":"ContainerDied","Data":"985167dacf7b8075ef0b18950fdbb5bbbecb7cd0882a556bd97af4e391aaeeea"}
Nov 03 07:30:20.646460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.646427    4470 generic.go:356] "Generic (PLEG): container finished" podID="bf996b97-74f9-450d-9ca9-826560308eb2" containerID="d59a2c6470550efbd2406309c0124f8a4803a5c89e7b299aff594a5b0053fec7" exitCode=60
Nov 03 07:30:20.646460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.646453    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" event={"ID":"bf996b97-74f9-450d-9ca9-826560308eb2","Type":"ContainerDied","Data":"d59a2c6470550efbd2406309c0124f8a4803a5c89e7b299aff594a5b0053fec7"}
Nov 03 07:30:20.646866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.646849    4470 scope.go:122] "RemoveContainer" containerID="d59a2c6470550efbd2406309c0124f8a4803a5c89e7b299aff594a5b0053fec7"
Nov 03 07:30:20.647291 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:20.647241    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8_odh-ai-gateway-infra(bf996b97-74f9-450d-9ca9-826560308eb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" podUID="bf996b97-74f9-450d-9ca9-826560308eb2"
Nov 03 07:30:20.654467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.654118    4470 scope.go:122] "RemoveContainer" containerID="0bf16c2ba46a709869f5f1cb619b42b620e6826bb7373b74822ed03c5762735b"
Nov 03 07:30:20.667376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.667350    4470 scope.go:122] "RemoveContainer" containerID="7adad0ba287f048d32e0068f7d26756879f6a6bee3def5d28795c648523b71f3"
Nov 03 07:30:20.681077 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.681049    4470 scope.go:122] "RemoveContainer" containerID="7fdd9c262f9f5ac52d5e7b8b6a38b64b1700598b74a4e730c5c075172453e0ae"
Nov 03 07:30:20.799524 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.799435    4470 scope.go:122] "RemoveContainer" containerID="e8b455440407cdd68f593ff92d1cae1d224ac2cddfa6454c07e7255f115950b8"
Nov 03 07:30:20.814144 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.814089    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"]
Nov 03 07:30:20.876477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:20.876446    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"
Nov 03 07:30:21.022809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.022771    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3cb7b51-48dc-4368-98a3-640982b6e68e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3cb7b51-48dc-4368-98a3-640982b6e68e-maas-api-service-ca\") pod \"e3cb7b51-48dc-4368-98a3-640982b6e68e\" (UID: \"e3cb7b51-48dc-4368-98a3-640982b6e68e\") "
Nov 03 07:30:21.023058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.023015    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3cb7b51-48dc-4368-98a3-640982b6e68e-kube-api-access-5b9lz\" (UniqueName: \"kubernetes.io/projected/e3cb7b51-48dc-4368-98a3-640982b6e68e-kube-api-access-5b9lz\") pod \"e3cb7b51-48dc-4368-98a3-640982b6e68e\" (UID: \"e3cb7b51-48dc-4368-98a3-640982b6e68e\") "
Nov 03 07:30:21.023255 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.023179    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cb7b51-48dc-4368-98a3-640982b6e68e-maas-api-service-ca" pod "e3cb7b51-48dc-4368-98a3-640982b6e68e" (UID: "e3cb7b51-48dc-4368-98a3-640982b6e68e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:30:21.023698 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.023675    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3cb7b51-48dc-4368-98a3-640982b6e68e-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:21.025502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.025472    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cb7b51-48dc-4368-98a3-640982b6e68e-kube-api-access-5b9lz" pod "e3cb7b51-48dc-4368-98a3-640982b6e68e" (UID: "e3cb7b51-48dc-4368-98a3-640982b6e68e"). InnerVolumeSpecName "kube-api-access-5b9lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:30:21.125785 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.125745    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b9lz\" (UniqueName: \"kubernetes.io/projected/e3cb7b51-48dc-4368-98a3-640982b6e68e-kube-api-access-5b9lz\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:21.139171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.139126    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"]
Nov 03 07:30:21.143441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.143416    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-09fd61-29894850-xxhsb"]
Nov 03 07:30:21.256489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.256452    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"]
Nov 03 07:30:21.262297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.262271    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-893d83-29894850-tchg2"]
Nov 03 07:30:21.430749 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.430656    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"]
Nov 03 07:30:21.661843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.661817    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"
Nov 03 07:30:21.661843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.661833    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw" event={"ID":"e3cb7b51-48dc-4368-98a3-640982b6e68e","Type":"ContainerDied","Data":"cef8ca137ac9a552c5c0199dbabf9389c21e5e5c0618f730b7503bde22501aaa"}
Nov 03 07:30:21.662111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.661902    4470 scope.go:122] "RemoveContainer" containerID="d9a3260382e434f05a4b35eec670df1cd855a55a1994ed57f9bf5912e97f0a4e"
Nov 03 07:30:21.669193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.669106    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" event={"ID":"de7a5180-e91f-4bec-815a-a870c63c3992","Type":"ContainerStarted","Data":"6d262d2f5e26bcb1ec9b4166e68f57cff48c07c42d69dc34a60a74afd82d499e"}
Nov 03 07:30:21.813153 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.812692    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838" path="/var/lib/kubelet/pods/07659463-2d5f-4171-9e7b-d3b83cd4b838/volumes"
Nov 03 07:30:21.816464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.813691    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" path="/var/lib/kubelet/pods/8e89e6f9-f776-4d44-9643-23c2d7e2922b/volumes"
Nov 03 07:30:21.816464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.814542    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69" path="/var/lib/kubelet/pods/d32f79ba-6fb3-447b-b224-639f835f6d69/volumes"
Nov 03 07:30:21.816464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.815719    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150" path="/var/lib/kubelet/pods/dae496e2-15ee-4ac2-8de0-b61d610f0150/volumes"
Nov 03 07:30:21.944621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.944591    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"
Nov 03 07:30:21.950310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:21.950282    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"
Nov 03 07:30:22.045354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.045275    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf996b97-74f9-450d-9ca9-826560308eb2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf996b97-74f9-450d-9ca9-826560308eb2-maas-api-service-ca\") pod \"bf996b97-74f9-450d-9ca9-826560308eb2\" (UID: \"bf996b97-74f9-450d-9ca9-826560308eb2\") "
Nov 03 07:30:22.045501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.045489    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10559d66-8937-43fe-9f78-953042895f8a-kube-api-access-xxxbl\" (UniqueName: \"kubernetes.io/projected/10559d66-8937-43fe-9f78-953042895f8a-kube-api-access-xxxbl\") pod \"10559d66-8937-43fe-9f78-953042895f8a\" (UID: \"10559d66-8937-43fe-9f78-953042895f8a\") "
Nov 03 07:30:22.045584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.045573    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf996b97-74f9-450d-9ca9-826560308eb2-kube-api-access-g9ntx\" (UniqueName: \"kubernetes.io/projected/bf996b97-74f9-450d-9ca9-826560308eb2-kube-api-access-g9ntx\") pod \"bf996b97-74f9-450d-9ca9-826560308eb2\" (UID: \"bf996b97-74f9-450d-9ca9-826560308eb2\") "
Nov 03 07:30:22.045632 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.045616    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10559d66-8937-43fe-9f78-953042895f8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10559d66-8937-43fe-9f78-953042895f8a-maas-api-service-ca\") pod \"10559d66-8937-43fe-9f78-953042895f8a\" (UID: \"10559d66-8937-43fe-9f78-953042895f8a\") "
Nov 03 07:30:22.045690 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.045658    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf996b97-74f9-450d-9ca9-826560308eb2-maas-api-service-ca" pod "bf996b97-74f9-450d-9ca9-826560308eb2" (UID: "bf996b97-74f9-450d-9ca9-826560308eb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:30:22.045982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.045960    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10559d66-8937-43fe-9f78-953042895f8a-maas-api-service-ca" pod "10559d66-8937-43fe-9f78-953042895f8a" (UID: "10559d66-8937-43fe-9f78-953042895f8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:30:22.046239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.046221    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10559d66-8937-43fe-9f78-953042895f8a-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:22.046295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.046244    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf996b97-74f9-450d-9ca9-826560308eb2-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:22.047899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.047878    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf996b97-74f9-450d-9ca9-826560308eb2-kube-api-access-g9ntx" pod "bf996b97-74f9-450d-9ca9-826560308eb2" (UID: "bf996b97-74f9-450d-9ca9-826560308eb2"). InnerVolumeSpecName "kube-api-access-g9ntx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:30:22.047978 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.047907    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10559d66-8937-43fe-9f78-953042895f8a-kube-api-access-xxxbl" pod "10559d66-8937-43fe-9f78-953042895f8a" (UID: "10559d66-8937-43fe-9f78-953042895f8a"). InnerVolumeSpecName "kube-api-access-xxxbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:30:22.080092 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.080049    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"]
Nov 03 07:30:22.088857 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.088825    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3fea3a-29894850-qb9gw"]
Nov 03 07:30:22.148023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.147988    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxxbl\" (UniqueName: \"kubernetes.io/projected/10559d66-8937-43fe-9f78-953042895f8a-kube-api-access-xxxbl\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:22.148023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.148017    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9ntx\" (UniqueName: \"kubernetes.io/projected/bf996b97-74f9-450d-9ca9-826560308eb2-kube-api-access-g9ntx\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:22.677733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.677700    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"
Nov 03 07:30:22.677733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.677712    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2" event={"ID":"10559d66-8937-43fe-9f78-953042895f8a","Type":"ContainerDied","Data":"a4be657c03675b69f477af8f169eb7afbfe73fc8ed23034c46b74159be951858"}
Nov 03 07:30:22.677989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.677779    4470 scope.go:122] "RemoveContainer" containerID="5268acc0bf7c1d3f952416798459d99a8f1c84ac5abf314f33e1176ffd547cfa"
Nov 03 07:30:22.680091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.680066    4470 generic.go:356] "Generic (PLEG): container finished" podID="de7a5180-e91f-4bec-815a-a870c63c3992" containerID="6d262d2f5e26bcb1ec9b4166e68f57cff48c07c42d69dc34a60a74afd82d499e" exitCode=60
Nov 03 07:30:22.680232 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.680120    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" event={"ID":"de7a5180-e91f-4bec-815a-a870c63c3992","Type":"ContainerDied","Data":"6d262d2f5e26bcb1ec9b4166e68f57cff48c07c42d69dc34a60a74afd82d499e"}
Nov 03 07:30:22.680542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.680526    4470 scope.go:122] "RemoveContainer" containerID="6d262d2f5e26bcb1ec9b4166e68f57cff48c07c42d69dc34a60a74afd82d499e"
Nov 03 07:30:22.680859 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:22.680830    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894850-56g2c_odh-ai-gateway-infra(de7a5180-e91f-4bec-815a-a870c63c3992)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" podUID="de7a5180-e91f-4bec-815a-a870c63c3992"
Nov 03 07:30:22.682942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.682919    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8" event={"ID":"bf996b97-74f9-450d-9ca9-826560308eb2","Type":"ContainerDied","Data":"1fcd935ae83629c56895c3cd72dc05614e4634a4f2e7d82bbcb3d278f464dde1"}
Nov 03 07:30:22.682942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.682939    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"
Nov 03 07:30:22.689939 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.689922    4470 scope.go:122] "RemoveContainer" containerID="e8b455440407cdd68f593ff92d1cae1d224ac2cddfa6454c07e7255f115950b8"
Nov 03 07:30:22.704483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.704466    4470 scope.go:122] "RemoveContainer" containerID="d59a2c6470550efbd2406309c0124f8a4803a5c89e7b299aff594a5b0053fec7"
Nov 03 07:30:22.851627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.851590    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"]
Nov 03 07:30:22.855163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.855117    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894850-7mhz2"]
Nov 03 07:30:22.969484 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.969397    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"]
Nov 03 07:30:22.977546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:22.977506    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894850-qxnj8"]
Nov 03 07:30:23.220309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:23.220230    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"]
Nov 03 07:30:23.831159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:23.823079    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10559d66-8937-43fe-9f78-953042895f8a" path="/var/lib/kubelet/pods/10559d66-8937-43fe-9f78-953042895f8a/volumes"
Nov 03 07:30:23.831159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:23.823778    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf996b97-74f9-450d-9ca9-826560308eb2" path="/var/lib/kubelet/pods/bf996b97-74f9-450d-9ca9-826560308eb2/volumes"
Nov 03 07:30:23.831159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:23.824801    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e" path="/var/lib/kubelet/pods/e3cb7b51-48dc-4368-98a3-640982b6e68e/volumes"
Nov 03 07:30:23.886356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:23.886330    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"
Nov 03 07:30:23.976094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:23.976020    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7a5180-e91f-4bec-815a-a870c63c3992-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7a5180-e91f-4bec-815a-a870c63c3992-maas-api-service-ca\") pod \"de7a5180-e91f-4bec-815a-a870c63c3992\" (UID: \"de7a5180-e91f-4bec-815a-a870c63c3992\") "
Nov 03 07:30:23.976298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:23.976261    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7a5180-e91f-4bec-815a-a870c63c3992-kube-api-access-5qppx\" (UniqueName: \"kubernetes.io/projected/de7a5180-e91f-4bec-815a-a870c63c3992-kube-api-access-5qppx\") pod \"de7a5180-e91f-4bec-815a-a870c63c3992\" (UID: \"de7a5180-e91f-4bec-815a-a870c63c3992\") "
Nov 03 07:30:23.976381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:23.976365    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7a5180-e91f-4bec-815a-a870c63c3992-maas-api-service-ca" pod "de7a5180-e91f-4bec-815a-a870c63c3992" (UID: "de7a5180-e91f-4bec-815a-a870c63c3992"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:30:23.976903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:23.976883    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7a5180-e91f-4bec-815a-a870c63c3992-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:23.978505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:23.978482    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7a5180-e91f-4bec-815a-a870c63c3992-kube-api-access-5qppx" pod "de7a5180-e91f-4bec-815a-a870c63c3992" (UID: "de7a5180-e91f-4bec-815a-a870c63c3992"). InnerVolumeSpecName "kube-api-access-5qppx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:30:24.078207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:24.078160    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qppx\" (UniqueName: \"kubernetes.io/projected/de7a5180-e91f-4bec-815a-a870c63c3992-kube-api-access-5qppx\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:30:24.703298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:24.703266    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c" event={"ID":"de7a5180-e91f-4bec-815a-a870c63c3992","Type":"ContainerDied","Data":"a026f58ba68f1ed55ca20f54a876112f9fe120376a1044b6e5a1d333e020d170"}
Nov 03 07:30:24.703478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:24.703323    4470 scope.go:122] "RemoveContainer" containerID="6d262d2f5e26bcb1ec9b4166e68f57cff48c07c42d69dc34a60a74afd82d499e"
Nov 03 07:30:24.703478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:24.703278    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"
Nov 03 07:30:24.880529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:24.880492    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"]
Nov 03 07:30:24.886903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:24.886875    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894850-56g2c"]
Nov 03 07:30:25.808202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:25.808150    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7a5180-e91f-4bec-815a-a870c63c3992" path="/var/lib/kubelet/pods/de7a5180-e91f-4bec-815a-a870c63c3992/volumes"
Nov 03 07:30:34.854838 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:30:34.854799    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:30:37.824157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:37.823231    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:30:37.862158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:37.859262    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 07:30:37.878354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:37.878310    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:30:37.889594 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:37.889265    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:30:37.929229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:37.913568    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:30:37.990172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:37.986251    4470 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:30:37.993700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:37.993641    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:30:37.994455 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:37.994427    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:30:38.114318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:38.114214    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Nov 03 07:30:38.134162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:38.133219    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc" podUID="ff22aa3b-6c14-43ef-8658-1d6a6d13c53c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 07:30:38.240215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:38.238219    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:30:46.019436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:46.019404    4470 generic.go:356] "Generic (PLEG): container finished" podID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerID="e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba" exitCode=0
Nov 03 07:30:46.019872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:46.019457    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" event={"ID":"c3c5935f-9218-450b-a7d5-8fd56685d917","Type":"ContainerDied","Data":"e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba"}
Nov 03 07:30:46.019915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:46.019883    4470 scope.go:122] "RemoveContainer" containerID="e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba"
Nov 03 07:30:46.021655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:46.021623    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:30:46.021750 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:46.021696    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:30:46.021750 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:46.021728    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:30:47.028542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:47.028508    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" event={"ID":"c3c5935f-9218-450b-a7d5-8fd56685d917","Type":"ContainerStarted","Data":"5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2"}
Nov 03 07:30:47.606392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:47.606360    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:30:47.606637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:47.606612    4470 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 07:30:47.606693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:47.606671    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 07:30:48.606852 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:48.606815    4470 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 07:30:48.607307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:48.606868    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 07:30:49.606642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:49.606599    4470 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 07:30:49.606836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:49.606667    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 07:30:50.607447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:50.607415    4470 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 07:30:50.607838 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:50.607470    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 07:30:51.606919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:51.606883    4470 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 07:30:51.607277 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:51.606932    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 07:30:52.610771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:52.610741    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:30:52.611284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:52.611264    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:30:52.612043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:52.612026    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:30:53.058371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:53.058294    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"]
Nov 03 07:30:53.061770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:53.061748    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18734a2e-nw6zq"]
Nov 03 07:30:53.808082 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:30:53.808051    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8939c53-836d-4a76-9cec-5ec2bb3afc95" path="/var/lib/kubelet/pods/d8939c53-836d-4a76-9cec-5ec2bb3afc95/volumes"
Nov 03 07:31:04.761286 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:31:04.761250    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:31:09.491478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.491427    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"]
Nov 03 07:31:09.493085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.493054    4470 cpu_manager.go:414] "containerMap: removing container" podUID="de7a5180-e91f-4bec-815a-a870c63c3992" containerName="cleanup"
Nov 03 07:31:09.493289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.493273    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7a5180-e91f-4bec-815a-a870c63c3992" containerName="cleanup"
Nov 03 07:31:09.493391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.493380    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerName="cleanup"
Nov 03 07:31:09.493480 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.493470    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerName="cleanup"
Nov 03 07:31:09.493559 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.493549    4470 cpu_manager.go:414] "containerMap: removing container" podUID="bf996b97-74f9-450d-9ca9-826560308eb2" containerName="cleanup"
Nov 03 07:31:09.493635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.493624    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf996b97-74f9-450d-9ca9-826560308eb2" containerName="cleanup"
Nov 03 07:31:09.493717 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.493708    4470 cpu_manager.go:414] "containerMap: removing container" podUID="10559d66-8937-43fe-9f78-953042895f8a" containerName="cleanup"
Nov 03 07:31:09.493799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.493788    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10559d66-8937-43fe-9f78-953042895f8a" containerName="cleanup"
Nov 03 07:31:09.493886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.493876    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerName="cleanup"
Nov 03 07:31:09.493971 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.493961    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerName="cleanup"
Nov 03 07:31:09.494049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494039    4470 cpu_manager.go:414] "containerMap: removing container" podUID="10559d66-8937-43fe-9f78-953042895f8a" containerName="cleanup"
Nov 03 07:31:09.494150 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494119    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10559d66-8937-43fe-9f78-953042895f8a" containerName="cleanup"
Nov 03 07:31:09.494266 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494252    4470 cpu_manager.go:414] "containerMap: removing container" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerName="cleanup"
Nov 03 07:31:09.494348 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494336    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerName="cleanup"
Nov 03 07:31:09.494426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494419    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerName="cleanup"
Nov 03 07:31:09.494481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494474    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerName="cleanup"
Nov 03 07:31:09.494531 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494525    4470 cpu_manager.go:414] "containerMap: removing container" podUID="de7a5180-e91f-4bec-815a-a870c63c3992" containerName="cleanup"
Nov 03 07:31:09.494579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494572    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7a5180-e91f-4bec-815a-a870c63c3992" containerName="cleanup"
Nov 03 07:31:09.494631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494624    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerName="cleanup"
Nov 03 07:31:09.494677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494671    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerName="cleanup"
Nov 03 07:31:09.494727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494721    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerName="cleanup"
Nov 03 07:31:09.494776 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494770    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerName="cleanup"
Nov 03 07:31:09.494827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494821    4470 cpu_manager.go:414] "containerMap: removing container" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerName="cleanup"
Nov 03 07:31:09.494872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494866    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerName="cleanup"
Nov 03 07:31:09.494927 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494921    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerName="cleanup"
Nov 03 07:31:09.494974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.494968    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerName="cleanup"
Nov 03 07:31:09.495018 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495012    4470 cpu_manager.go:414] "containerMap: removing container" podUID="bf996b97-74f9-450d-9ca9-826560308eb2" containerName="cleanup"
Nov 03 07:31:09.495062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495056    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf996b97-74f9-450d-9ca9-826560308eb2" containerName="cleanup"
Nov 03 07:31:09.495113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495107    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerName="cleanup"
Nov 03 07:31:09.495195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495185    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerName="cleanup"
Nov 03 07:31:09.495260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495209    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerName="cleanup"
Nov 03 07:31:09.495260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495221    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerName="cleanup"
Nov 03 07:31:09.495260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495257    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerName="cleanup"
Nov 03 07:31:09.495408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495264    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerName="cleanup"
Nov 03 07:31:09.495408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495281    4470 cpu_manager.go:414] "containerMap: removing container" podUID="bf996b97-74f9-450d-9ca9-826560308eb2" containerName="cleanup"
Nov 03 07:31:09.495408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495289    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf996b97-74f9-450d-9ca9-826560308eb2" containerName="cleanup"
Nov 03 07:31:09.495408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495303    4470 cpu_manager.go:414] "containerMap: removing container" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerName="cleanup"
Nov 03 07:31:09.495408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495310    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerName="cleanup"
Nov 03 07:31:09.495408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495328    4470 cpu_manager.go:414] "containerMap: removing container" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerName="cleanup"
Nov 03 07:31:09.495408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495336    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495521    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495535    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495545    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7a5180-e91f-4bec-815a-a870c63c3992" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495553    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495562    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495570    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495578    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495588    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf996b97-74f9-450d-9ca9-826560308eb2" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495603    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="10559d66-8937-43fe-9f78-953042895f8a" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495612    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495620    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7a5180-e91f-4bec-815a-a870c63c3992" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495630    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf996b97-74f9-450d-9ca9-826560308eb2" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495651    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495662    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495673    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495683    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495692    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerName="cleanup"
Nov 03 07:31:09.495759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.495701    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="10559d66-8937-43fe-9f78-953042895f8a" containerName="cleanup"
Nov 03 07:31:09.521170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.514722    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:31:09.538644 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.538611    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"]
Nov 03 07:31:09.543182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.543153    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8c9423-852c-40f5-b69d-6338dc0d310f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz\" (UID: \"3f8c9423-852c-40f5-b69d-6338dc0d310f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:31:09.543501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.543479    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4km5\" (UniqueName: \"kubernetes.io/projected/3f8c9423-852c-40f5-b69d-6338dc0d310f-kube-api-access-r4km5\") pod \"payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz\" (UID: \"3f8c9423-852c-40f5-b69d-6338dc0d310f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:31:09.581491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.577548    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-32b500-dockercfg-gb67s\""
Nov 03 07:31:09.581491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.580802    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-32b500\""
Nov 03 07:31:09.645638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.645603    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8c9423-852c-40f5-b69d-6338dc0d310f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz\" (UID: \"3f8c9423-852c-40f5-b69d-6338dc0d310f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:31:09.645808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.645747    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4km5\" (UniqueName: \"kubernetes.io/projected/3f8c9423-852c-40f5-b69d-6338dc0d310f-kube-api-access-r4km5\") pod \"payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz\" (UID: \"3f8c9423-852c-40f5-b69d-6338dc0d310f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:31:09.646389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.646371    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8c9423-852c-40f5-b69d-6338dc0d310f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz\" (UID: \"3f8c9423-852c-40f5-b69d-6338dc0d310f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:31:09.712290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.712259    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4km5\" (UniqueName: \"kubernetes.io/projected/3f8c9423-852c-40f5-b69d-6338dc0d310f-kube-api-access-r4km5\") pod \"payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz\" (UID: \"3f8c9423-852c-40f5-b69d-6338dc0d310f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:31:09.850010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:09.849980    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:31:10.269304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:10.269273    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"]
Nov 03 07:31:11.258836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:11.258799    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz" event={"ID":"3f8c9423-852c-40f5-b69d-6338dc0d310f","Type":"ContainerStarted","Data":"b16cdb1d107a543281314a3f2e8566f573574a11fbf4f8d67d307d7e78cfb0f2"}
Nov 03 07:31:11.259340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:11.258858    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz" event={"ID":"3f8c9423-852c-40f5-b69d-6338dc0d310f","Type":"ContainerStarted","Data":"60086db41c6371996bd390bfd09bfb550b87f3ad683619b9f59a68d6527cdf01"}
Nov 03 07:31:11.259340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:11.258914    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:31:11.380322 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:11.380263    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz" podStartSLOduration=2.38024845 podStartE2EDuration="2.38024845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:11.379113492 +0000 UTC m=+1923.711482477" watchObservedRunningTime="2026-11-03 07:31:11.38024845 +0000 UTC m=+1923.712617483"
Nov 03 07:31:13.601161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:13.593260    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"]
Nov 03 07:31:13.601161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:13.593513    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerName="istio-proxy" containerID="cri-o://ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26" gracePeriod=30
Nov 03 07:31:13.724180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:13.724120    4470 scope.go:122] "RemoveContainer" containerID="343123871b70f0202ed8214c50528569918b0e7a4f83d227e5c219e6d607c9c1"
Nov 03 07:31:13.737098 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:13.737069    4470 scope.go:122] "RemoveContainer" containerID="cbc3e3b62cf654a47aa9e5624ba7e5225629b81344925928510b3ec61f8f0f96"
Nov 03 07:31:13.749462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:13.749440    4470 scope.go:122] "RemoveContainer" containerID="db05ee991b656051438b13ad53f1d15c9393030b305058387725e78a357208b5"
Nov 03 07:31:14.901715 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.901680    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"]
Nov 03 07:31:14.902413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902396    4470 cpu_manager.go:414] "containerMap: removing container" podUID="10559d66-8937-43fe-9f78-953042895f8a" containerName="cleanup"
Nov 03 07:31:14.902413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902413    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10559d66-8937-43fe-9f78-953042895f8a" containerName="cleanup"
Nov 03 07:31:14.902516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902430    4470 cpu_manager.go:414] "containerMap: removing container" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerName="cleanup"
Nov 03 07:31:14.902516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902435    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerName="cleanup"
Nov 03 07:31:14.902516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902447    4470 cpu_manager.go:414] "containerMap: removing container" podUID="de7a5180-e91f-4bec-815a-a870c63c3992" containerName="cleanup"
Nov 03 07:31:14.902516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902452    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7a5180-e91f-4bec-815a-a870c63c3992" containerName="cleanup"
Nov 03 07:31:14.902516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902460    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerName="cleanup"
Nov 03 07:31:14.902516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902466    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerName="cleanup"
Nov 03 07:31:14.902516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902474    4470 cpu_manager.go:414] "containerMap: removing container" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerName="cleanup"
Nov 03 07:31:14.902516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902479    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerName="cleanup"
Nov 03 07:31:14.902516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902491    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerName="cleanup"
Nov 03 07:31:14.902516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902497    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerName="cleanup"
Nov 03 07:31:14.902883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902533    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerName="cleanup"
Nov 03 07:31:14.902883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902537    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerName="cleanup"
Nov 03 07:31:14.902883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902633    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e505ebba-8ed9-4532-988d-5d3ff724b522" containerName="cleanup"
Nov 03 07:31:14.902883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902640    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7a5180-e91f-4bec-815a-a870c63c3992" containerName="cleanup"
Nov 03 07:31:14.902883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902649    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf996b97-74f9-450d-9ca9-826560308eb2" containerName="cleanup"
Nov 03 07:31:14.902883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902658    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="07659463-2d5f-4171-9e7b-d3b83cd4b838" containerName="cleanup"
Nov 03 07:31:14.902883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902665    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e89e6f9-f776-4d44-9643-23c2d7e2922b" containerName="cleanup"
Nov 03 07:31:14.902883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902672    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="10559d66-8937-43fe-9f78-953042895f8a" containerName="cleanup"
Nov 03 07:31:14.902883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902678    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae496e2-15ee-4ac2-8de0-b61d610f0150" containerName="cleanup"
Nov 03 07:31:14.902883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902684    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3cb7b51-48dc-4368-98a3-640982b6e68e" containerName="cleanup"
Nov 03 07:31:14.902883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.902692    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32f79ba-6fb3-447b-b224-639f835f6d69" containerName="cleanup"
Nov 03 07:31:14.908335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.908314    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:14.989669 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:14.976678    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-72a642-openshift-default-dockercfg-z5kmq\""
Nov 03 07:31:15.018896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.018854    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-certs\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.018896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.018908    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlb4c\" (UniqueName: \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-kube-api-access-vlb4c\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.019229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.019019    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-credential-socket\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.019300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.019228    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-podinfo\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.019300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.019277    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-data\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.019411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.019321    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-token\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.019411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.019369    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-socket\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.019411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.019392    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4854012-52e4-4b86-81e9-3af28f40a8a1-istiod-ca-cert\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.019587 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.019460    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-envoy\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.019587 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.019563    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"]
Nov 03 07:31:15.121877 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.121840    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-certs\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.121877 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.121886    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlb4c\" (UniqueName: \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-kube-api-access-vlb4c\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.121910    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-credential-socket\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.121988    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-podinfo\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.122014    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-data\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.122051    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-token\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.122100    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-socket\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.122123    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4854012-52e4-4b86-81e9-3af28f40a8a1-istiod-ca-cert\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.122213    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-envoy\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.122320    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-certs\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.122371    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-credential-socket\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.122436    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-data\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.122629    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-socket\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.122898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.122876    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4854012-52e4-4b86-81e9-3af28f40a8a1-istiod-ca-cert\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.124754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.124722    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-envoy\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.124754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.124743    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-podinfo\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.186588 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.186513    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlb4c\" (UniqueName: \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-kube-api-access-vlb4c\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.186736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.186633    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-token\") pod \"e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") " pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.245512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.229579    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:15.828483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.828445    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"]
Nov 03 07:31:15.855478 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:31:15.855446    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4854012_52e4_4b86_81e9_3af28f40a8a1.slice/crio-344b01f1503830cccc3abd3427cbe098aef7557c2269d4c98e48c7bea48a6d44 WatchSource:0}: Error finding container 344b01f1503830cccc3abd3427cbe098aef7557c2269d4c98e48c7bea48a6d44: Status 404 returned error can't find the container with id 344b01f1503830cccc3abd3427cbe098aef7557c2269d4c98e48c7bea48a6d44
Nov 03 07:31:15.858092 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.858057    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:31:15.858203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.858156    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:31:15.858245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:15.858211    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:31:16.328675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:16.328633    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" event={"ID":"c4854012-52e4-4b86-81e9-3af28f40a8a1","Type":"ContainerStarted","Data":"3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388"}
Nov 03 07:31:16.329366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:16.328696    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" event={"ID":"c4854012-52e4-4b86-81e9-3af28f40a8a1","Type":"ContainerStarted","Data":"344b01f1503830cccc3abd3427cbe098aef7557c2269d4c98e48c7bea48a6d44"}
Nov 03 07:31:16.505122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:16.505048    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" podStartSLOduration=2.505033558 podStartE2EDuration="2.505033558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:16.503432371 +0000 UTC m=+1928.835801378" watchObservedRunningTime="2026-11-03 07:31:16.505033558 +0000 UTC m=+1928.837402552"
Nov 03 07:31:17.232217 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:17.232179    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:17.237769 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:17.237742    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:17.336464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:17.336430    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:17.337571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:17.337552    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:31:18.564941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.564859    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"]
Nov 03 07:31:18.569469 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.569449    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:18.675866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.675827    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:31:18.676011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.675874    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:31:18.676011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.675874    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:31:18.676204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.676103    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:31:18.680276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.680255    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"]
Nov 03 07:31:18.700510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.700475    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a271ff-glzf2\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:18.700738 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.700530    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x4m4\" (UniqueName: \"kubernetes.io/projected/34e4621a-80bc-4277-a35c-503e84e16875-kube-api-access-2x4m4\") pod \"maas-api-revoke-sub-06a271ff-glzf2\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:18.700880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.700794    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a271ff-glzf2\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:18.805244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.802572    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a271ff-glzf2\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:18.805244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.802625    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x4m4\" (UniqueName: \"kubernetes.io/projected/34e4621a-80bc-4277-a35c-503e84e16875-kube-api-access-2x4m4\") pod \"maas-api-revoke-sub-06a271ff-glzf2\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:18.805244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.802797    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a271ff-glzf2\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:18.805244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.804384    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a271ff-glzf2\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:18.836070 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.821652    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a271ff-glzf2\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:18.870014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.869949    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x4m4\" (UniqueName: \"kubernetes.io/projected/34e4621a-80bc-4277-a35c-503e84e16875-kube-api-access-2x4m4\") pod \"maas-api-revoke-sub-06a271ff-glzf2\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:18.884910 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.884874    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:18.927261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:18.927232    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:31:19.010387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.006722    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-socket\") pod \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") "
Nov 03 07:31:19.010387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.006881    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-kube-api-access-8545v\" (UniqueName: \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-kube-api-access-8545v\") pod \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") "
Nov 03 07:31:19.010387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.006985    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istiod-ca-cert\") pod \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") "
Nov 03 07:31:19.012288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.010836    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-credential-socket\") pod \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") "
Nov 03 07:31:19.012288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.011023    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-podinfo\") pod \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") "
Nov 03 07:31:19.012288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.011072    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-envoy\") pod \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") "
Nov 03 07:31:19.012288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.011149    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-certs\") pod \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") "
Nov 03 07:31:19.012288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.011183    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-token\" (UniqueName: \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-token\") pod \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") "
Nov 03 07:31:19.012288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.011298    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-data\") pod \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\" (UID: \"284f7fe9-4542-47c4-bb83-ff3b25260bcd\") "
Nov 03 07:31:19.027405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.019353    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-certs" pod "284f7fe9-4542-47c4-bb83-ff3b25260bcd" (UID: "284f7fe9-4542-47c4-bb83-ff3b25260bcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:31:19.027405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.019719    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-credential-socket" pod "284f7fe9-4542-47c4-bb83-ff3b25260bcd" (UID: "284f7fe9-4542-47c4-bb83-ff3b25260bcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:31:19.027405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.020034    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istiod-ca-cert" pod "284f7fe9-4542-47c4-bb83-ff3b25260bcd" (UID: "284f7fe9-4542-47c4-bb83-ff3b25260bcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:31:19.040173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.039152    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-socket" pod "284f7fe9-4542-47c4-bb83-ff3b25260bcd" (UID: "284f7fe9-4542-47c4-bb83-ff3b25260bcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:31:19.065159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.041266    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-data" pod "284f7fe9-4542-47c4-bb83-ff3b25260bcd" (UID: "284f7fe9-4542-47c4-bb83-ff3b25260bcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:31:19.077159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.065907    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:19.077159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.065944    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:19.077159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.065960    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:19.077159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.065974    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:19.077159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.065989    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:19.247983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.247894    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-token" pod "284f7fe9-4542-47c4-bb83-ff3b25260bcd" (UID: "284f7fe9-4542-47c4-bb83-ff3b25260bcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:31:19.253392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.251941    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-kube-api-access-8545v" pod "284f7fe9-4542-47c4-bb83-ff3b25260bcd" (UID: "284f7fe9-4542-47c4-bb83-ff3b25260bcd"). InnerVolumeSpecName "kube-api-access-8545v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:31:19.253392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.252030    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-envoy" pod "284f7fe9-4542-47c4-bb83-ff3b25260bcd" (UID: "284f7fe9-4542-47c4-bb83-ff3b25260bcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:31:19.264826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.264770    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-podinfo" pod "284f7fe9-4542-47c4-bb83-ff3b25260bcd" (UID: "284f7fe9-4542-47c4-bb83-ff3b25260bcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:31:19.270420 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.270324    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:19.270420 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.270359    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:19.270420 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.270374    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:19.270420 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.270388    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8545v\" (UniqueName: \"kubernetes.io/projected/284f7fe9-4542-47c4-bb83-ff3b25260bcd-kube-api-access-8545v\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:19.356770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.356729    4470 generic.go:356] "Generic (PLEG): container finished" podID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerID="ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26" exitCode=0
Nov 03 07:31:19.357121 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.357097    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"
Nov 03 07:31:19.357477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.357457    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" event={"ID":"284f7fe9-4542-47c4-bb83-ff3b25260bcd","Type":"ContainerDied","Data":"ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26"}
Nov 03 07:31:19.357523 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.357502    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf" event={"ID":"284f7fe9-4542-47c4-bb83-ff3b25260bcd","Type":"ContainerDied","Data":"1222b8e5074139b6c79d7796e1d642b71cd62c3e44a800ae53f0412a927ac392"}
Nov 03 07:31:19.357556 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.357530    4470 scope.go:122] "RemoveContainer" containerID="ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26"
Nov 03 07:31:19.374002 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.373978    4470 scope.go:122] "RemoveContainer" containerID="ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26"
Nov 03 07:31:19.374346 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:31:19.374325    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26\": container with ID starting with ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26 not found: ID does not exist" containerID="ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26"
Nov 03 07:31:19.374402 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.374355    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26"} err="failed to get container status \"ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26\": rpc error: code = NotFound desc = could not find container \"ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26\": container with ID starting with ed6d14f66db152bb10841072f67a0cb550ab9f0c443c19d0cfef2d77811b4b26 not found: ID does not exist"
Nov 03 07:31:19.487090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.487059    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"]
Nov 03 07:31:19.536786 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.536755    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"]
Nov 03 07:31:19.543879 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.543852    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf"]
Nov 03 07:31:19.810780 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:19.810710    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" path="/var/lib/kubelet/pods/284f7fe9-4542-47c4-bb83-ff3b25260bcd/volumes"
Nov 03 07:31:20.370794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:20.370757    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2" event={"ID":"34e4621a-80bc-4277-a35c-503e84e16875","Type":"ContainerStarted","Data":"8d14a6c535542d16aa328423b3f11676a287ad5e47a89dc5dac05963ddfab466"}
Nov 03 07:31:20.371017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:20.370815    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2" event={"ID":"34e4621a-80bc-4277-a35c-503e84e16875","Type":"ContainerStarted","Data":"6c861b4c53932d9a25fc8536f73ec525bfc8f7d1e75956c0e8ae955ba4876082"}
Nov 03 07:31:20.550158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:20.526030    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2" podStartSLOduration=2.526008853 podStartE2EDuration="2.526008853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:20.525365122 +0000 UTC m=+1932.857734118" watchObservedRunningTime="2026-11-03 07:31:20.526008853 +0000 UTC m=+1932.858377849"
Nov 03 07:31:21.379536 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:21.379502    4470 generic.go:356] "Generic (PLEG): container finished" podID="34e4621a-80bc-4277-a35c-503e84e16875" containerID="8d14a6c535542d16aa328423b3f11676a287ad5e47a89dc5dac05963ddfab466" exitCode=0
Nov 03 07:31:21.379961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:21.379589    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2" event={"ID":"34e4621a-80bc-4277-a35c-503e84e16875","Type":"ContainerDied","Data":"8d14a6c535542d16aa328423b3f11676a287ad5e47a89dc5dac05963ddfab466"}
Nov 03 07:31:21.907184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:21.907115    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"]
Nov 03 07:31:21.908099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:21.908076    4470 cpu_manager.go:414] "containerMap: removing container" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerName="istio-proxy"
Nov 03 07:31:21.908099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:21.908096    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerName="istio-proxy"
Nov 03 07:31:21.908308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:21.908266    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="284f7fe9-4542-47c4-bb83-ff3b25260bcd" containerName="istio-proxy"
Nov 03 07:31:21.912395 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:21.912372    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:31:21.968385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:21.968113    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-1fa30f\""
Nov 03 07:31:21.968711 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:21.968687    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"]
Nov 03 07:31:22.042158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.021380    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r99xn\" (UniqueName: \"kubernetes.io/projected/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-kube-api-access-r99xn\") pod \"payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b\" (UID: \"091fa31f-5c4c-4817-bc3d-0c36edcac9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:31:22.042158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.021445    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b\" (UID: \"091fa31f-5c4c-4817-bc3d-0c36edcac9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:31:22.042158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.021901    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-1fa30f-dockercfg-zw6d6\""
Nov 03 07:31:22.136753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.136718    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99xn\" (UniqueName: \"kubernetes.io/projected/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-kube-api-access-r99xn\") pod \"payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b\" (UID: \"091fa31f-5c4c-4817-bc3d-0c36edcac9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:31:22.137049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.137013    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b\" (UID: \"091fa31f-5c4c-4817-bc3d-0c36edcac9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:31:22.138471 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.138446    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b\" (UID: \"091fa31f-5c4c-4817-bc3d-0c36edcac9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:31:22.174236 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.174098    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"]
Nov 03 07:31:22.196657 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.196625    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"]
Nov 03 07:31:22.197047 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.197034    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:31:22.207212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.207183    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r99xn\" (UniqueName: \"kubernetes.io/projected/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-kube-api-access-r99xn\") pod \"payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b\" (UID: \"091fa31f-5c4c-4817-bc3d-0c36edcac9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:31:22.230164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.229740    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:31:22.238986 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.238952    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk\" (UID: \"a8b9a208-a748-4e5f-a83e-b075f08bd3a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:31:22.239126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.239105    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcr65\" (UniqueName: \"kubernetes.io/projected/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-kube-api-access-rcr65\") pod \"payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk\" (UID: \"a8b9a208-a748-4e5f-a83e-b075f08bd3a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:31:22.270928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.270458    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:31:22.341656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.341388    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk\" (UID: \"a8b9a208-a748-4e5f-a83e-b075f08bd3a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:31:22.341656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.341579    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcr65\" (UniqueName: \"kubernetes.io/projected/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-kube-api-access-rcr65\") pod \"payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk\" (UID: \"a8b9a208-a748-4e5f-a83e-b075f08bd3a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:31:22.342795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.342766    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk\" (UID: \"a8b9a208-a748-4e5f-a83e-b075f08bd3a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:31:22.405757 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.405727    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcr65\" (UniqueName: \"kubernetes.io/projected/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-kube-api-access-rcr65\") pod \"payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk\" (UID: \"a8b9a208-a748-4e5f-a83e-b075f08bd3a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:31:22.546745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.546718    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:31:22.561176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.561100    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:22.629365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.627122    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"]
Nov 03 07:31:22.646660 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:31:22.646621    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091fa31f_5c4c_4817_bc3d_0c36edcac9c3.slice/crio-3f4fe8f9fa457792f06c02f9532826f8e19b51267d0326f2eda535537b5c9ae4 WatchSource:0}: Error finding container 3f4fe8f9fa457792f06c02f9532826f8e19b51267d0326f2eda535537b5c9ae4: Status 404 returned error can't find the container with id 3f4fe8f9fa457792f06c02f9532826f8e19b51267d0326f2eda535537b5c9ae4
Nov 03 07:31:22.647593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.647420    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e4621a-80bc-4277-a35c-503e84e16875-kube-api-access-2x4m4\" (UniqueName: \"kubernetes.io/projected/34e4621a-80bc-4277-a35c-503e84e16875-kube-api-access-2x4m4\") pod \"34e4621a-80bc-4277-a35c-503e84e16875\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") "
Nov 03 07:31:22.648093 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.647661    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-openshift-ca\") pod \"34e4621a-80bc-4277-a35c-503e84e16875\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") "
Nov 03 07:31:22.648333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.648050    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-openshift-ca" pod "34e4621a-80bc-4277-a35c-503e84e16875" (UID: "34e4621a-80bc-4277-a35c-503e84e16875"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:31:22.653074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.653048    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-xks-ca\") pod \"34e4621a-80bc-4277-a35c-503e84e16875\" (UID: \"34e4621a-80bc-4277-a35c-503e84e16875\") "
Nov 03 07:31:22.654480 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.653999    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:22.663026 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.662999    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-xks-ca" pod "34e4621a-80bc-4277-a35c-503e84e16875" (UID: "34e4621a-80bc-4277-a35c-503e84e16875"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:31:22.673465 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.673428    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e4621a-80bc-4277-a35c-503e84e16875-kube-api-access-2x4m4" pod "34e4621a-80bc-4277-a35c-503e84e16875" (UID: "34e4621a-80bc-4277-a35c-503e84e16875"). InnerVolumeSpecName "kube-api-access-2x4m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:31:22.755119 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.755082    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x4m4\" (UniqueName: \"kubernetes.io/projected/34e4621a-80bc-4277-a35c-503e84e16875-kube-api-access-2x4m4\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:22.755119 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.755122    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34e4621a-80bc-4277-a35c-503e84e16875-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:31:22.993493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:22.993462    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"]
Nov 03 07:31:22.994025 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:31:22.993998    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b9a208_a748_4e5f_a83e_b075f08bd3a2.slice/crio-d9f9b583f1dfc79ee166fb3074407de925f1d8ad22a3f4331676780cf128e205 WatchSource:0}: Error finding container d9f9b583f1dfc79ee166fb3074407de925f1d8ad22a3f4331676780cf128e205: Status 404 returned error can't find the container with id d9f9b583f1dfc79ee166fb3074407de925f1d8ad22a3f4331676780cf128e205
Nov 03 07:31:23.398751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:23.398699    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b" event={"ID":"091fa31f-5c4c-4817-bc3d-0c36edcac9c3","Type":"ContainerStarted","Data":"795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45"}
Nov 03 07:31:23.398952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:23.398783    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b" event={"ID":"091fa31f-5c4c-4817-bc3d-0c36edcac9c3","Type":"ContainerStarted","Data":"3f4fe8f9fa457792f06c02f9532826f8e19b51267d0326f2eda535537b5c9ae4"}
Nov 03 07:31:23.398952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:23.398815    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:31:23.400190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:23.400164    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2" event={"ID":"34e4621a-80bc-4277-a35c-503e84e16875","Type":"ContainerDied","Data":"6c861b4c53932d9a25fc8536f73ec525bfc8f7d1e75956c0e8ae955ba4876082"}
Nov 03 07:31:23.400297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:23.400189    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"
Nov 03 07:31:23.400297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:23.400215    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c861b4c53932d9a25fc8536f73ec525bfc8f7d1e75956c0e8ae955ba4876082"
Nov 03 07:31:23.402229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:23.402200    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk" event={"ID":"a8b9a208-a748-4e5f-a83e-b075f08bd3a2","Type":"ContainerStarted","Data":"6af27f1014f3c2f66022674c116102f4390592dbeb1d23b03e4d131642551911"}
Nov 03 07:31:23.402330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:23.402240    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk" event={"ID":"a8b9a208-a748-4e5f-a83e-b075f08bd3a2","Type":"ContainerStarted","Data":"d9f9b583f1dfc79ee166fb3074407de925f1d8ad22a3f4331676780cf128e205"}
Nov 03 07:31:23.402396 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:23.402347    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:31:23.549181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:23.549098    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b" podStartSLOduration=2.549078063 podStartE2EDuration="2.549078063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:23.545400569 +0000 UTC m=+1935.877769564" watchObservedRunningTime="2026-11-03 07:31:23.549078063 +0000 UTC m=+1935.881447054"
Nov 03 07:31:23.729677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:23.729572    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk" podStartSLOduration=2.729558531 podStartE2EDuration="2.729558531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:23.727810163 +0000 UTC m=+1936.060179157" watchObservedRunningTime="2026-11-03 07:31:23.729558531 +0000 UTC m=+1936.061927525"
Nov 03 07:31:26.887901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:26.887862    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"]
Nov 03 07:31:26.888881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:26.888858    4470 cpu_manager.go:414] "containerMap: removing container" podUID="34e4621a-80bc-4277-a35c-503e84e16875" containerName="revoke-keys"
Nov 03 07:31:26.888881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:26.888881    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e4621a-80bc-4277-a35c-503e84e16875" containerName="revoke-keys"
Nov 03 07:31:26.889203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:26.889128    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e4621a-80bc-4277-a35c-503e84e16875" containerName="revoke-keys"
Nov 03 07:31:26.893436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:26.893415    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:31:26.954327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:26.954299    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"]
Nov 03 07:31:27.134162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.121940    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-72a642\""
Nov 03 07:31:27.134162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.122078    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-72a642-dockercfg-w767f\""
Nov 03 07:31:27.134162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.125936    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wl6r\" (UniqueName: \"kubernetes.io/projected/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-kube-api-access-9wl6r\") pod \"payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs\" (UID: \"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:31:27.134162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.126180    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs\" (UID: \"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:31:27.142203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.140857    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"]
Nov 03 07:31:27.230053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.229456    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:31:27.230053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.229635    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs\" (UID: \"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:31:27.230053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.229806    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wl6r\" (UniqueName: \"kubernetes.io/projected/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-kube-api-access-9wl6r\") pod \"payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs\" (UID: \"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:31:27.231216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.231117    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs\" (UID: \"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:31:27.236824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.236801    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"]
Nov 03 07:31:27.293616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.293587    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wl6r\" (UniqueName: \"kubernetes.io/projected/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-kube-api-access-9wl6r\") pod \"payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs\" (UID: \"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:31:27.330776 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.330738    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwp6c\" (UniqueName: \"kubernetes.io/projected/26d59cce-9220-4573-abb1-a17d6bedc1c9-kube-api-access-zwp6c\") pod \"payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm\" (UID: \"26d59cce-9220-4573-abb1-a17d6bedc1c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:31:27.330776 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.330779    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d59cce-9220-4573-abb1-a17d6bedc1c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm\" (UID: \"26d59cce-9220-4573-abb1-a17d6bedc1c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:31:27.432181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.432063    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwp6c\" (UniqueName: \"kubernetes.io/projected/26d59cce-9220-4573-abb1-a17d6bedc1c9-kube-api-access-zwp6c\") pod \"payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm\" (UID: \"26d59cce-9220-4573-abb1-a17d6bedc1c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:31:27.432181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.432107    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d59cce-9220-4573-abb1-a17d6bedc1c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm\" (UID: \"26d59cce-9220-4573-abb1-a17d6bedc1c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:31:27.432877 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.432857    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d59cce-9220-4573-abb1-a17d6bedc1c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm\" (UID: \"26d59cce-9220-4573-abb1-a17d6bedc1c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:31:27.500207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.500170    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwp6c\" (UniqueName: \"kubernetes.io/projected/26d59cce-9220-4573-abb1-a17d6bedc1c9-kube-api-access-zwp6c\") pod \"payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm\" (UID: \"26d59cce-9220-4573-abb1-a17d6bedc1c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:31:27.509937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.509909    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:31:27.550294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.550258    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:31:27.898311 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:31:27.898282    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a85e2c_44c2_4b66_9ae4_ee6da8ad4495.slice/crio-148013a8a4557c066d236d1b91f97af33908846e2e508006a1fda0948a2b9e3b WatchSource:0}: Error finding container 148013a8a4557c066d236d1b91f97af33908846e2e508006a1fda0948a2b9e3b: Status 404 returned error can't find the container with id 148013a8a4557c066d236d1b91f97af33908846e2e508006a1fda0948a2b9e3b
Nov 03 07:31:27.899368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.899345    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"]
Nov 03 07:31:27.962317 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:31:27.962291    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d59cce_9220_4573_abb1_a17d6bedc1c9.slice/crio-10fa226a822e15e084952ff94f4483963d9125770f39791d1cb907d968d50f10 WatchSource:0}: Error finding container 10fa226a822e15e084952ff94f4483963d9125770f39791d1cb907d968d50f10: Status 404 returned error can't find the container with id 10fa226a822e15e084952ff94f4483963d9125770f39791d1cb907d968d50f10
Nov 03 07:31:27.963089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:27.963067    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"]
Nov 03 07:31:28.456177 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:28.456121    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs" event={"ID":"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495","Type":"ContainerStarted","Data":"4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9"}
Nov 03 07:31:28.456379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:28.456196    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs" event={"ID":"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495","Type":"ContainerStarted","Data":"148013a8a4557c066d236d1b91f97af33908846e2e508006a1fda0948a2b9e3b"}
Nov 03 07:31:28.456379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:28.456223    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:31:28.457923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:28.457888    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm" event={"ID":"26d59cce-9220-4573-abb1-a17d6bedc1c9","Type":"ContainerStarted","Data":"3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb"}
Nov 03 07:31:28.458043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:28.457942    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:31:28.458043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:28.457968    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm" event={"ID":"26d59cce-9220-4573-abb1-a17d6bedc1c9","Type":"ContainerStarted","Data":"10fa226a822e15e084952ff94f4483963d9125770f39791d1cb907d968d50f10"}
Nov 03 07:31:28.578906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:28.578855    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs" podStartSLOduration=2.5788417839999997 podStartE2EDuration="2.578841784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:28.575233005 +0000 UTC m=+1940.907602002" watchObservedRunningTime="2026-11-03 07:31:28.578841784 +0000 UTC m=+1940.911210819"
Nov 03 07:31:28.701870 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:28.701793    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm" podStartSLOduration=2.7017548099999997 podStartE2EDuration="2.70175481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:28.696663749 +0000 UTC m=+1941.029032744" watchObservedRunningTime="2026-11-03 07:31:28.70175481 +0000 UTC m=+1941.034123808"
Nov 03 07:31:34.542880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:34.542845    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:31:34.544892 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:34.544028    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:31:34.761191 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:31:34.761122    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:31:39.477526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:39.477498    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:31:39.477947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:39.477579    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:31:58.309931 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:58.309873    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"]
Nov 03 07:31:58.310509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:31:58.310156    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy" containerID="cri-o://72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1" gracePeriod=30
Nov 03 07:32:03.612862 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.612827    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:32:03.693468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.693438    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-credential-socket\") pod \"2fc30b05-f8f1-4c5c-9197-384960da56e9\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") "
Nov 03 07:32:03.693672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.693496    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-envoy\") pod \"2fc30b05-f8f1-4c5c-9197-384960da56e9\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") "
Nov 03 07:32:03.693672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.693528    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-socket\") pod \"2fc30b05-f8f1-4c5c-9197-384960da56e9\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") "
Nov 03 07:32:03.693672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.693568    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-certs\") pod \"2fc30b05-f8f1-4c5c-9197-384960da56e9\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") "
Nov 03 07:32:03.693672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.693610    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc30b05-f8f1-4c5c-9197-384960da56e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc30b05-f8f1-4c5c-9197-384960da56e9-istiod-ca-cert\") pod \"2fc30b05-f8f1-4c5c-9197-384960da56e9\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") "
Nov 03 07:32:03.693889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.693767    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-credential-socket" pod "2fc30b05-f8f1-4c5c-9197-384960da56e9" (UID: "2fc30b05-f8f1-4c5c-9197-384960da56e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:32:03.693889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.693800    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-socket" pod "2fc30b05-f8f1-4c5c-9197-384960da56e9" (UID: "2fc30b05-f8f1-4c5c-9197-384960da56e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:32:03.693889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.693839    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-kube-api-access-fzxl8\" (UniqueName: \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-kube-api-access-fzxl8\") pod \"2fc30b05-f8f1-4c5c-9197-384960da56e9\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") "
Nov 03 07:32:03.694062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.693907    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-token\" (UniqueName: \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-token\") pod \"2fc30b05-f8f1-4c5c-9197-384960da56e9\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") "
Nov 03 07:32:03.694062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.693937    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-data\") pod \"2fc30b05-f8f1-4c5c-9197-384960da56e9\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") "
Nov 03 07:32:03.694062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.694046    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-podinfo\") pod \"2fc30b05-f8f1-4c5c-9197-384960da56e9\" (UID: \"2fc30b05-f8f1-4c5c-9197-384960da56e9\") "
Nov 03 07:32:03.694261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.693933    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-certs" pod "2fc30b05-f8f1-4c5c-9197-384960da56e9" (UID: "2fc30b05-f8f1-4c5c-9197-384960da56e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:32:03.694261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.694060    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc30b05-f8f1-4c5c-9197-384960da56e9-istiod-ca-cert" pod "2fc30b05-f8f1-4c5c-9197-384960da56e9" (UID: "2fc30b05-f8f1-4c5c-9197-384960da56e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:32:03.694366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.694259    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-data" pod "2fc30b05-f8f1-4c5c-9197-384960da56e9" (UID: "2fc30b05-f8f1-4c5c-9197-384960da56e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:32:03.695077 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.695049    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:03.695225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.695080    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:03.695225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.695095    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:03.695225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.695110    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc30b05-f8f1-4c5c-9197-384960da56e9-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:03.695225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.695124    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:03.696307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.696284    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-envoy" pod "2fc30b05-f8f1-4c5c-9197-384960da56e9" (UID: "2fc30b05-f8f1-4c5c-9197-384960da56e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:32:03.696399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.696353    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-kube-api-access-fzxl8" pod "2fc30b05-f8f1-4c5c-9197-384960da56e9" (UID: "2fc30b05-f8f1-4c5c-9197-384960da56e9"). InnerVolumeSpecName "kube-api-access-fzxl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:32:03.696469 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.696439    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-token" pod "2fc30b05-f8f1-4c5c-9197-384960da56e9" (UID: "2fc30b05-f8f1-4c5c-9197-384960da56e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:32:03.696512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.696465    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-podinfo" pod "2fc30b05-f8f1-4c5c-9197-384960da56e9" (UID: "2fc30b05-f8f1-4c5c-9197-384960da56e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:32:03.797351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.797297    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:03.797351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.797339    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzxl8\" (UniqueName: \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-kube-api-access-fzxl8\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:03.797351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.797354    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:03.797679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.797368    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc30b05-f8f1-4c5c-9197-384960da56e9-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:03.858402 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.858319    4470 generic.go:356] "Generic (PLEG): container finished" podID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerID="72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1" exitCode=0
Nov 03 07:32:03.858402 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.858371    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"
Nov 03 07:32:03.858621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.858413    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" event={"ID":"2fc30b05-f8f1-4c5c-9197-384960da56e9","Type":"ContainerDied","Data":"72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1"}
Nov 03 07:32:03.858621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.858465    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79" event={"ID":"2fc30b05-f8f1-4c5c-9197-384960da56e9","Type":"ContainerDied","Data":"2010e1e6e3b7c4806a3be45befb61f4cf4053a1545d31cd9e3789a5b689a4446"}
Nov 03 07:32:03.858621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.858492    4470 scope.go:122] "RemoveContainer" containerID="72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1"
Nov 03 07:32:03.870105 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.870086    4470 scope.go:122] "RemoveContainer" containerID="72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1"
Nov 03 07:32:03.870457 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:32:03.870440    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1\": container with ID starting with 72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1 not found: ID does not exist" containerID="72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1"
Nov 03 07:32:03.870504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:03.870465    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1"} err="failed to get container status \"72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1\": rpc error: code = NotFound desc = could not find container \"72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1\": container with ID starting with 72affd32d1dcea68e64f92bc21e4998b36add49caa44884038db5572019771e1 not found: ID does not exist"
Nov 03 07:32:04.033915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:04.033124    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"]
Nov 03 07:32:04.038621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:04.038590    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79"]
Nov 03 07:32:04.762348 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:32:04.762313    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:32:05.809773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:05.809740    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" path="/var/lib/kubelet/pods/2fc30b05-f8f1-4c5c-9197-384960da56e9/volumes"
Nov 03 07:32:09.627819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.627787    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"]
Nov 03 07:32:09.628583 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.628568    4470 cpu_manager.go:414] "containerMap: removing container" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy"
Nov 03 07:32:09.628643 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.628585    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy"
Nov 03 07:32:09.628749 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.628738    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc30b05-f8f1-4c5c-9197-384960da56e9" containerName="istio-proxy"
Nov 03 07:32:09.634333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.634306    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:09.691076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.691044    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"]
Nov 03 07:32:09.696260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.696233    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klt9j\" (UniqueName: \"kubernetes.io/projected/0c7b2070-63f2-4492-aa27-feda9aecadd5-kube-api-access-klt9j\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:09.696417 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.696400    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:09.696490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.696434    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:09.737087 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.737054    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:32:09.737087 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.737051    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:32:09.739414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.739378    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:32:09.739558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.739443    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:32:09.798555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.798518    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klt9j\" (UniqueName: \"kubernetes.io/projected/0c7b2070-63f2-4492-aa27-feda9aecadd5-kube-api-access-klt9j\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:09.798732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.798711    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:09.798774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.798763    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:09.799472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.799449    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:09.801403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.801378    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:09.862746 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.862701    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klt9j\" (UniqueName: \"kubernetes.io/projected/0c7b2070-63f2-4492-aa27-feda9aecadd5-kube-api-access-klt9j\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:09.949559 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:09.949480    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:10.334622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:10.334595    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"]
Nov 03 07:32:10.335161 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:32:10.335100    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7b2070_63f2_4492_aa27_feda9aecadd5.slice/crio-8062f1053eca1eb0774c32bd6e27db1a175f422ec16e80a5a07f2ef9401ea4b6 WatchSource:0}: Error finding container 8062f1053eca1eb0774c32bd6e27db1a175f422ec16e80a5a07f2ef9401ea4b6: Status 404 returned error can't find the container with id 8062f1053eca1eb0774c32bd6e27db1a175f422ec16e80a5a07f2ef9401ea4b6
Nov 03 07:32:10.917593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:10.917488    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7" event={"ID":"0c7b2070-63f2-4492-aa27-feda9aecadd5","Type":"ContainerStarted","Data":"c311e571f461b4ea040b5b26005667f0b445b3466bb08ed9db6c4b3d37d15a1b"}
Nov 03 07:32:10.917593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:10.917554    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7" event={"ID":"0c7b2070-63f2-4492-aa27-feda9aecadd5","Type":"ContainerStarted","Data":"8062f1053eca1eb0774c32bd6e27db1a175f422ec16e80a5a07f2ef9401ea4b6"}
Nov 03 07:32:11.039231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:11.039171    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7" podStartSLOduration=2.039129009 podStartE2EDuration="2.039129009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:32:11.038309862 +0000 UTC m=+1983.370678870" watchObservedRunningTime="2026-11-03 07:32:11.039129009 +0000 UTC m=+1983.371524988"
Nov 03 07:32:11.925791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:11.925752    4470 generic.go:356] "Generic (PLEG): container finished" podID="0c7b2070-63f2-4492-aa27-feda9aecadd5" containerID="c311e571f461b4ea040b5b26005667f0b445b3466bb08ed9db6c4b3d37d15a1b" exitCode=0
Nov 03 07:32:11.926244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:11.925859    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7" event={"ID":"0c7b2070-63f2-4492-aa27-feda9aecadd5","Type":"ContainerDied","Data":"c311e571f461b4ea040b5b26005667f0b445b3466bb08ed9db6c4b3d37d15a1b"}
Nov 03 07:32:13.115344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.115316    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:13.254672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.254597    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-openshift-ca\") pod \"0c7b2070-63f2-4492-aa27-feda9aecadd5\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") "
Nov 03 07:32:13.254840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.254699    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-xks-ca\") pod \"0c7b2070-63f2-4492-aa27-feda9aecadd5\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") "
Nov 03 07:32:13.254840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.254807    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7b2070-63f2-4492-aa27-feda9aecadd5-kube-api-access-klt9j\" (UniqueName: \"kubernetes.io/projected/0c7b2070-63f2-4492-aa27-feda9aecadd5-kube-api-access-klt9j\") pod \"0c7b2070-63f2-4492-aa27-feda9aecadd5\" (UID: \"0c7b2070-63f2-4492-aa27-feda9aecadd5\") "
Nov 03 07:32:13.255094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.255065    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-openshift-ca" pod "0c7b2070-63f2-4492-aa27-feda9aecadd5" (UID: "0c7b2070-63f2-4492-aa27-feda9aecadd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:32:13.255822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.255791    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:13.257066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.257039    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-xks-ca" pod "0c7b2070-63f2-4492-aa27-feda9aecadd5" (UID: "0c7b2070-63f2-4492-aa27-feda9aecadd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:32:13.257185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.257122    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7b2070-63f2-4492-aa27-feda9aecadd5-kube-api-access-klt9j" pod "0c7b2070-63f2-4492-aa27-feda9aecadd5" (UID: "0c7b2070-63f2-4492-aa27-feda9aecadd5"). InnerVolumeSpecName "kube-api-access-klt9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:32:13.357784 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.357750    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7b2070-63f2-4492-aa27-feda9aecadd5-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:13.357784 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.357783    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klt9j\" (UniqueName: \"kubernetes.io/projected/0c7b2070-63f2-4492-aa27-feda9aecadd5-kube-api-access-klt9j\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:13.834778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.834749    4470 scope.go:122] "RemoveContainer" containerID="332c69d355fb1f703ce9d2f836642595a1fb9506b38b15091e8a8ba5cfe9b559"
Nov 03 07:32:13.845921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.845900    4470 scope.go:122] "RemoveContainer" containerID="f31c61f8a669e6af6aabde545c97a88ec2fed180796dae43976dc20f22ddd79c"
Nov 03 07:32:13.942844 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.942808    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7" event={"ID":"0c7b2070-63f2-4492-aa27-feda9aecadd5","Type":"ContainerDied","Data":"8062f1053eca1eb0774c32bd6e27db1a175f422ec16e80a5a07f2ef9401ea4b6"}
Nov 03 07:32:13.943022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.942858    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"
Nov 03 07:32:13.943022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:13.942863    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8062f1053eca1eb0774c32bd6e27db1a175f422ec16e80a5a07f2ef9401ea4b6"
Nov 03 07:32:22.077638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:22.077604    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"]
Nov 03 07:32:27.084308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:27.083964    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt" podUID="4062b59b-02fe-4ed1-b3a8-4b20c296c6ee" containerName="payload-pre-processing" containerID="cri-o://88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217" gracePeriod=25
Nov 03 07:32:27.378369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:27.378338    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:32:27.518474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:27.518443    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-kube-api-access-kpqrs\" (UniqueName: \"kubernetes.io/projected/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-kube-api-access-kpqrs\") pod \"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee\" (UID: \"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee\") "
Nov 03 07:32:27.518805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:27.518771    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-plugins-config-volume\") pod \"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee\" (UID: \"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee\") "
Nov 03 07:32:27.519213 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:27.519189    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-plugins-config-volume" pod "4062b59b-02fe-4ed1-b3a8-4b20c296c6ee" (UID: "4062b59b-02fe-4ed1-b3a8-4b20c296c6ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:32:27.519448 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:27.519432    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:27.521205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:27.521178    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-kube-api-access-kpqrs" pod "4062b59b-02fe-4ed1-b3a8-4b20c296c6ee" (UID: "4062b59b-02fe-4ed1-b3a8-4b20c296c6ee"). InnerVolumeSpecName "kube-api-access-kpqrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:32:27.621468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:27.621385    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpqrs\" (UniqueName: \"kubernetes.io/projected/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee-kube-api-access-kpqrs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:32:28.062753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:28.062728    4470 generic.go:356] "Generic (PLEG): container finished" podID="4062b59b-02fe-4ed1-b3a8-4b20c296c6ee" containerID="88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217" exitCode=0
Nov 03 07:32:28.062933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:28.062777    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"
Nov 03 07:32:28.062933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:28.062823    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt" event={"ID":"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee","Type":"ContainerDied","Data":"88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217"}
Nov 03 07:32:28.062933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:28.062884    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt" event={"ID":"4062b59b-02fe-4ed1-b3a8-4b20c296c6ee","Type":"ContainerDied","Data":"48d795b1e5f0513050342c747b329fabe4e852bce64419841875b38afeee91ce"}
Nov 03 07:32:28.062933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:28.062913    4470 scope.go:122] "RemoveContainer" containerID="88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217"
Nov 03 07:32:28.073777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:28.073756    4470 scope.go:122] "RemoveContainer" containerID="88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217"
Nov 03 07:32:28.074075 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:32:28.074051    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217\": container with ID starting with 88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217 not found: ID does not exist" containerID="88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217"
Nov 03 07:32:28.074251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:28.074080    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217"} err="failed to get container status \"88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217\": rpc error: code = NotFound desc = could not find container \"88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217\": container with ID starting with 88e97945134762b4b48caa2a3fa399a4757407dffbe5abbc0b530ec1c7960217 not found: ID does not exist"
Nov 03 07:32:28.238889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:28.238857    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"]
Nov 03 07:32:28.289313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:28.289282    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt"]
Nov 03 07:32:29.808555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:32:29.808516    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4062b59b-02fe-4ed1-b3a8-4b20c296c6ee" path="/var/lib/kubelet/pods/4062b59b-02fe-4ed1-b3a8-4b20c296c6ee/volumes"
Nov 03 07:32:34.761295 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:32:34.761262    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:33:01.841444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.841410    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"]
Nov 03 07:33:01.842202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.842185    4470 cpu_manager.go:414] "containerMap: removing container" podUID="4062b59b-02fe-4ed1-b3a8-4b20c296c6ee" containerName="payload-pre-processing"
Nov 03 07:33:01.842272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.842204    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4062b59b-02fe-4ed1-b3a8-4b20c296c6ee" containerName="payload-pre-processing"
Nov 03 07:33:01.842272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.842233    4470 cpu_manager.go:414] "containerMap: removing container" podUID="0c7b2070-63f2-4492-aa27-feda9aecadd5" containerName="revoke-keys"
Nov 03 07:33:01.842272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.842238    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7b2070-63f2-4492-aa27-feda9aecadd5" containerName="revoke-keys"
Nov 03 07:33:01.842428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.842351    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="4062b59b-02fe-4ed1-b3a8-4b20c296c6ee" containerName="payload-pre-processing"
Nov 03 07:33:01.842428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.842363    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7b2070-63f2-4492-aa27-feda9aecadd5" containerName="revoke-keys"
Nov 03 07:33:01.846083 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.846066    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.885801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.885763    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-data\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.886002 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.885870    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istiod-ca-cert\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.886002 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.885961    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-credential-socket\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.886125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.886004    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-token\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.886229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.886213    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-socket\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.886332 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.886318    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-certs\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.886391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.886350    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-podinfo\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.886450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.886390    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-envoy\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.886510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.886462    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hl6c\" (UniqueName: \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-kube-api-access-6hl6c\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.904154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.903832    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f6d9a2-openshift-default-dockercfg-fxm9q\""
Nov 03 07:33:01.958897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.958854    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"]
Nov 03 07:33:01.988255 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.988220    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-certs\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.988255 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.988268    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-podinfo\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.988565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.988308    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-envoy\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.988565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.988367    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hl6c\" (UniqueName: \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-kube-api-access-6hl6c\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.988565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.988473    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-data\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.988565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.988536    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istiod-ca-cert\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.988795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.988595    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-credential-socket\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.988795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.988627    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-token\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.988795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.988681    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-certs\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.988952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.988808    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-socket\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.989113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.989091    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-credential-socket\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.989440 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.989416    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-socket\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.989735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.989716    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istiod-ca-cert\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.990154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.990110    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-data\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.992768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.992744    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-envoy\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:01.992946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:01.992929    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-podinfo\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:02.053253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:02.053219    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-token\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:02.053428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:02.053408    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hl6c\" (UniqueName: \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-kube-api-access-6hl6c\") pod \"e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") " pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:02.161586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:02.161491    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:02.607418 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:33:02.607387    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c084dd_3680_4f60_80a1_a1b35e7a15ce.slice/crio-b881ad60df653b176126da40a2b6adf3173c779a32ee68f62c47da436fccc577 WatchSource:0}: Error finding container b881ad60df653b176126da40a2b6adf3173c779a32ee68f62c47da436fccc577: Status 404 returned error can't find the container with id b881ad60df653b176126da40a2b6adf3173c779a32ee68f62c47da436fccc577
Nov 03 07:33:02.607626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:02.607603    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"]
Nov 03 07:33:02.610176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:02.610106    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:33:02.610288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:02.610223    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:33:02.610352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:02.610290    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:33:03.437916 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:03.437875    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" event={"ID":"05c084dd-3680-4f60-80a1-a1b35e7a15ce","Type":"ContainerStarted","Data":"622390a52f6b3bbec04dea77086726ac2bbe536be61cdcf9922027aceca5088d"}
Nov 03 07:33:03.438385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:03.437944    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" event={"ID":"05c084dd-3680-4f60-80a1-a1b35e7a15ce","Type":"ContainerStarted","Data":"b881ad60df653b176126da40a2b6adf3173c779a32ee68f62c47da436fccc577"}
Nov 03 07:33:03.616514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:03.616448    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" podStartSLOduration=2.6164287489999998 podStartE2EDuration="2.616428749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:33:03.615370913 +0000 UTC m=+2035.947739909" watchObservedRunningTime="2026-11-03 07:33:03.616428749 +0000 UTC m=+2035.948797745"
Nov 03 07:33:04.162504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:04.162476    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:04.168583 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:04.168557    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:04.444983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:04.444901    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:04.445944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:04.445925    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:33:04.761850 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:33:04.761771    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:33:05.227871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.227840    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"]
Nov 03 07:33:05.232373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.232356    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:05.287969 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.287912    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:33:05.287969 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.287953    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:33:05.288209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.287994    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:33:05.289647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.289628    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"]
Nov 03 07:33:05.336113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.336083    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:33:05.351538 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.351495    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7mq\" (UniqueName: \"kubernetes.io/projected/f4ffac1f-d94c-4998-88b0-872da521e534-kube-api-access-wb7mq\") pod \"maas-api-revoke-keys-e2e-api-a-418f41-g5ngs\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:05.351734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.351605    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-418f41-g5ngs\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:05.351734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.351701    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-418f41-g5ngs\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:05.453435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.453399    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7mq\" (UniqueName: \"kubernetes.io/projected/f4ffac1f-d94c-4998-88b0-872da521e534-kube-api-access-wb7mq\") pod \"maas-api-revoke-keys-e2e-api-a-418f41-g5ngs\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:05.453917 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.453492    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-418f41-g5ngs\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:05.453917 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.453580    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-418f41-g5ngs\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:05.454221 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.454200    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-418f41-g5ngs\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:05.456184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.456160    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-418f41-g5ngs\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:05.522064 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.521986    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7mq\" (UniqueName: \"kubernetes.io/projected/f4ffac1f-d94c-4998-88b0-872da521e534-kube-api-access-wb7mq\") pod \"maas-api-revoke-keys-e2e-api-a-418f41-g5ngs\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:05.547821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.547794    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:05.928436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:05.928408    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"]
Nov 03 07:33:05.928571 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:33:05.928414    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ffac1f_d94c_4998_88b0_872da521e534.slice/crio-4a096b7b56a129357a5f0fac9b2324eb3b3db1476e1dbe4aa9dcec9c48954b01 WatchSource:0}: Error finding container 4a096b7b56a129357a5f0fac9b2324eb3b3db1476e1dbe4aa9dcec9c48954b01: Status 404 returned error can't find the container with id 4a096b7b56a129357a5f0fac9b2324eb3b3db1476e1dbe4aa9dcec9c48954b01
Nov 03 07:33:06.475771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:06.475733    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs" event={"ID":"f4ffac1f-d94c-4998-88b0-872da521e534","Type":"ContainerStarted","Data":"fa29d8323ac4a065ebe290d463c979c1651abf26c1de109da5f765c3f85fdeb1"}
Nov 03 07:33:06.476308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:06.475826    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs" event={"ID":"f4ffac1f-d94c-4998-88b0-872da521e534","Type":"ContainerStarted","Data":"4a096b7b56a129357a5f0fac9b2324eb3b3db1476e1dbe4aa9dcec9c48954b01"}
Nov 03 07:33:06.603695 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:06.603633    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs" podStartSLOduration=1.603618411 podStartE2EDuration="1.603618411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:33:06.599606242 +0000 UTC m=+2038.931975229" watchObservedRunningTime="2026-11-03 07:33:06.603618411 +0000 UTC m=+2038.935987454"
Nov 03 07:33:07.485907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:07.485858    4470 generic.go:356] "Generic (PLEG): container finished" podID="f4ffac1f-d94c-4998-88b0-872da521e534" containerID="fa29d8323ac4a065ebe290d463c979c1651abf26c1de109da5f765c3f85fdeb1" exitCode=0
Nov 03 07:33:07.486386 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:07.485985    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs" event={"ID":"f4ffac1f-d94c-4998-88b0-872da521e534","Type":"ContainerDied","Data":"fa29d8323ac4a065ebe290d463c979c1651abf26c1de109da5f765c3f85fdeb1"}
Nov 03 07:33:08.667611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:08.667589    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:08.808394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:08.808300    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-xks-ca\") pod \"f4ffac1f-d94c-4998-88b0-872da521e534\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") "
Nov 03 07:33:08.808570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:08.808547    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ffac1f-d94c-4998-88b0-872da521e534-kube-api-access-wb7mq\" (UniqueName: \"kubernetes.io/projected/f4ffac1f-d94c-4998-88b0-872da521e534-kube-api-access-wb7mq\") pod \"f4ffac1f-d94c-4998-88b0-872da521e534\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") "
Nov 03 07:33:08.808660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:08.808592    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-openshift-ca\") pod \"f4ffac1f-d94c-4998-88b0-872da521e534\" (UID: \"f4ffac1f-d94c-4998-88b0-872da521e534\") "
Nov 03 07:33:08.814152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:08.809037    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-openshift-ca" pod "f4ffac1f-d94c-4998-88b0-872da521e534" (UID: "f4ffac1f-d94c-4998-88b0-872da521e534"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:33:08.814152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:08.809457    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:33:08.814152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:08.810785    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-xks-ca" pod "f4ffac1f-d94c-4998-88b0-872da521e534" (UID: "f4ffac1f-d94c-4998-88b0-872da521e534"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:33:08.814152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:08.811344    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ffac1f-d94c-4998-88b0-872da521e534-kube-api-access-wb7mq" pod "f4ffac1f-d94c-4998-88b0-872da521e534" (UID: "f4ffac1f-d94c-4998-88b0-872da521e534"). InnerVolumeSpecName "kube-api-access-wb7mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:33:08.911331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:08.911294    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7mq\" (UniqueName: \"kubernetes.io/projected/f4ffac1f-d94c-4998-88b0-872da521e534-kube-api-access-wb7mq\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:33:08.911331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:08.911326    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ffac1f-d94c-4998-88b0-872da521e534-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:33:09.503929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:09.503904    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"
Nov 03 07:33:09.504111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:09.503944    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs" event={"ID":"f4ffac1f-d94c-4998-88b0-872da521e534","Type":"ContainerDied","Data":"4a096b7b56a129357a5f0fac9b2324eb3b3db1476e1dbe4aa9dcec9c48954b01"}
Nov 03 07:33:09.504111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:09.503999    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a096b7b56a129357a5f0fac9b2324eb3b3db1476e1dbe4aa9dcec9c48954b01"
Nov 03 07:33:13.345085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.345054    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"]
Nov 03 07:33:13.345882 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.345867    4470 cpu_manager.go:414] "containerMap: removing container" podUID="f4ffac1f-d94c-4998-88b0-872da521e534" containerName="revoke-keys"
Nov 03 07:33:13.345926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.345885    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ffac1f-d94c-4998-88b0-872da521e534" containerName="revoke-keys"
Nov 03 07:33:13.346034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.346024    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ffac1f-d94c-4998-88b0-872da521e534" containerName="revoke-keys"
Nov 03 07:33:13.349869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.349851    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:33:13.411207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.411172    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"]
Nov 03 07:33:13.464427 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.437348    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt\" (UID: \"ecc50dc9-fdea-459b-9cc1-db1aa822aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:33:13.493239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.489323    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm9ds\" (UniqueName: \"kubernetes.io/projected/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-kube-api-access-cm9ds\") pod \"payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt\" (UID: \"ecc50dc9-fdea-459b-9cc1-db1aa822aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:33:13.493239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.455307    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f6d9a2\""
Nov 03 07:33:13.515821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.515789    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f6d9a2-dockercfg-r6nfv\""
Nov 03 07:33:13.649505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.649423    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"]
Nov 03 07:33:13.657224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.657050    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt\" (UID: \"ecc50dc9-fdea-459b-9cc1-db1aa822aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:33:13.665307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.665275    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm9ds\" (UniqueName: \"kubernetes.io/projected/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-kube-api-access-cm9ds\") pod \"payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt\" (UID: \"ecc50dc9-fdea-459b-9cc1-db1aa822aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:33:13.670397 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.670259    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:33:13.672831 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.672783    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt\" (UID: \"ecc50dc9-fdea-459b-9cc1-db1aa822aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:33:13.682707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.679031    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"]
Nov 03 07:33:13.746661 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.746624    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm9ds\" (UniqueName: \"kubernetes.io/projected/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-kube-api-access-cm9ds\") pod \"payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt\" (UID: \"ecc50dc9-fdea-459b-9cc1-db1aa822aac9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:33:13.768775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.768741    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c19799-37af-4990-b6df-c1d93cafaf93-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k\" (UID: \"11c19799-37af-4990-b6df-c1d93cafaf93\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:33:13.768923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.768875    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rb7k\" (UniqueName: \"kubernetes.io/projected/11c19799-37af-4990-b6df-c1d93cafaf93-kube-api-access-5rb7k\") pod \"payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k\" (UID: \"11c19799-37af-4990-b6df-c1d93cafaf93\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:33:13.870808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.870768    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rb7k\" (UniqueName: \"kubernetes.io/projected/11c19799-37af-4990-b6df-c1d93cafaf93-kube-api-access-5rb7k\") pod \"payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k\" (UID: \"11c19799-37af-4990-b6df-c1d93cafaf93\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:33:13.871270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.871221    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c19799-37af-4990-b6df-c1d93cafaf93-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k\" (UID: \"11c19799-37af-4990-b6df-c1d93cafaf93\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:33:13.871909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.871887    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c19799-37af-4990-b6df-c1d93cafaf93-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k\" (UID: \"11c19799-37af-4990-b6df-c1d93cafaf93\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:33:13.936123 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.936092    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rb7k\" (UniqueName: \"kubernetes.io/projected/11c19799-37af-4990-b6df-c1d93cafaf93-kube-api-access-5rb7k\") pod \"payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k\" (UID: \"11c19799-37af-4990-b6df-c1d93cafaf93\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:33:13.980156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.979787    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:33:14.007162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:13.997684    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:33:14.572680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:14.572650    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"]
Nov 03 07:33:14.573007 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:33:14.572661    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc50dc9_fdea_459b_9cc1_db1aa822aac9.slice/crio-754f303132d3b1fdad0019f5c3be971dfe157be6f3049b9bc4ec972db97fed7a WatchSource:0}: Error finding container 754f303132d3b1fdad0019f5c3be971dfe157be6f3049b9bc4ec972db97fed7a: Status 404 returned error can't find the container with id 754f303132d3b1fdad0019f5c3be971dfe157be6f3049b9bc4ec972db97fed7a
Nov 03 07:33:14.620422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:14.620394    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"]
Nov 03 07:33:14.621634 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:33:14.621607    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c19799_37af_4990_b6df_c1d93cafaf93.slice/crio-978b6c92ae18031506ef7a0fa562a797ea4422f8da39b40c02cec34ece6a96f3 WatchSource:0}: Error finding container 978b6c92ae18031506ef7a0fa562a797ea4422f8da39b40c02cec34ece6a96f3: Status 404 returned error can't find the container with id 978b6c92ae18031506ef7a0fa562a797ea4422f8da39b40c02cec34ece6a96f3
Nov 03 07:33:14.675205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:14.675160    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k" event={"ID":"11c19799-37af-4990-b6df-c1d93cafaf93","Type":"ContainerStarted","Data":"978b6c92ae18031506ef7a0fa562a797ea4422f8da39b40c02cec34ece6a96f3"}
Nov 03 07:33:14.677975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:14.677492    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt" event={"ID":"ecc50dc9-fdea-459b-9cc1-db1aa822aac9","Type":"ContainerStarted","Data":"e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984"}
Nov 03 07:33:14.677975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:14.677547    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt" event={"ID":"ecc50dc9-fdea-459b-9cc1-db1aa822aac9","Type":"ContainerStarted","Data":"754f303132d3b1fdad0019f5c3be971dfe157be6f3049b9bc4ec972db97fed7a"}
Nov 03 07:33:14.678313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:14.678278    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:33:14.829218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:14.828266    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt" podStartSLOduration=1.8282458780000002 podStartE2EDuration="1.828245878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:33:14.816508036 +0000 UTC m=+2047.148877031" watchObservedRunningTime="2026-11-03 07:33:14.828245878 +0000 UTC m=+2047.160614874"
Nov 03 07:33:15.724157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:15.715111    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k" event={"ID":"11c19799-37af-4990-b6df-c1d93cafaf93","Type":"ContainerStarted","Data":"5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d"}
Nov 03 07:33:15.724157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:15.715503    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:33:15.874229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:15.872859    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k" podStartSLOduration=2.8728391 podStartE2EDuration="2.8728391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:33:15.870555269 +0000 UTC m=+2048.202924263" watchObservedRunningTime="2026-11-03 07:33:15.8728391 +0000 UTC m=+2048.205208095"
Nov 03 07:33:24.787818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:24.787726    4470 generic.go:356] "Generic (PLEG): container finished" podID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerID="3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388" exitCode=0
Nov 03 07:33:24.787818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:24.787801    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" event={"ID":"c4854012-52e4-4b86-81e9-3af28f40a8a1","Type":"ContainerDied","Data":"3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388"}
Nov 03 07:33:24.788390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:24.788270    4470 scope.go:122] "RemoveContainer" containerID="3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388"
Nov 03 07:33:24.790012 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:24.789980    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:33:24.790109 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:24.790050    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:33:24.790109 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:24.790084    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:33:25.719354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:25.719311    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:33:25.797535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:25.797499    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" event={"ID":"c4854012-52e4-4b86-81e9-3af28f40a8a1","Type":"ContainerStarted","Data":"55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c"}
Nov 03 07:33:26.232569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:26.232536    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:33:26.232755 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:26.232726    4470 patch_prober.go:29] interesting pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 07:33:26.232816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:26.232771    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 07:33:26.723578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:26.723551    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:33:27.232115 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:27.232083    4470 patch_prober.go:29] interesting pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 07:33:27.232511 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:27.232162    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 07:33:28.077991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.077950    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"]
Nov 03 07:33:28.082808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.082784    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:28.129968 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.129938    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"]
Nov 03 07:33:28.176159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.175865    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:33:28.176159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.175973    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:33:28.176159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.176021    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:33:28.187476 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.187454    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:33:28.205699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.205671    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf88ec81-cz6xn\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:28.205864 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.205728    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf88ec81-cz6xn\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:28.206000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.205979    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4hf\" (UniqueName: \"kubernetes.io/projected/962feb66-3458-4d86-a7f7-690c4192c916-kube-api-access-db4hf\") pod \"maas-api-revoke-sub-cf88ec81-cz6xn\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:28.232542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.232511    4470 patch_prober.go:29] interesting pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 07:33:28.232917 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.232566    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 07:33:28.307602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.307566    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4hf\" (UniqueName: \"kubernetes.io/projected/962feb66-3458-4d86-a7f7-690c4192c916-kube-api-access-db4hf\") pod \"maas-api-revoke-sub-cf88ec81-cz6xn\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:28.307781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.307727    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf88ec81-cz6xn\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:28.307781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.307772    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf88ec81-cz6xn\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:28.308434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.308406    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf88ec81-cz6xn\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:28.310352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.310329    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf88ec81-cz6xn\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:28.376328 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.376243    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4hf\" (UniqueName: \"kubernetes.io/projected/962feb66-3458-4d86-a7f7-690c4192c916-kube-api-access-db4hf\") pod \"maas-api-revoke-sub-cf88ec81-cz6xn\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:28.397460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.397429    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:28.877659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:28.877626    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"]
Nov 03 07:33:29.232167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:29.232100    4470 patch_prober.go:29] interesting pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 07:33:29.232354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:29.232204    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 07:33:29.829305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:29.829268    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn" event={"ID":"962feb66-3458-4d86-a7f7-690c4192c916","Type":"ContainerStarted","Data":"5ae0c70df59fca01e9611c590730b96a28d7cba509a5914362dcdb9187cd55a7"}
Nov 03 07:33:29.829743 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:29.829321    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn" event={"ID":"962feb66-3458-4d86-a7f7-690c4192c916","Type":"ContainerStarted","Data":"34ee81ded5e320f7fb037dcd7d131ea317018fbf258ca5cc4a409797f68b7892"}
Nov 03 07:33:29.961165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:29.950918    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn" podStartSLOduration=1.95089954 podStartE2EDuration="1.95089954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:33:29.95056341 +0000 UTC m=+2062.282932405" watchObservedRunningTime="2026-11-03 07:33:29.95089954 +0000 UTC m=+2062.283268535"
Nov 03 07:33:30.232761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:30.232678    4470 patch_prober.go:29] interesting pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 07:33:30.232761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:30.232742    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 07:33:30.839997 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:30.839964    4470 generic.go:356] "Generic (PLEG): container finished" podID="962feb66-3458-4d86-a7f7-690c4192c916" containerID="5ae0c70df59fca01e9611c590730b96a28d7cba509a5914362dcdb9187cd55a7" exitCode=0
Nov 03 07:33:30.840432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:30.840055    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn" event={"ID":"962feb66-3458-4d86-a7f7-690c4192c916","Type":"ContainerDied","Data":"5ae0c70df59fca01e9611c590730b96a28d7cba509a5914362dcdb9187cd55a7"}
Nov 03 07:33:31.232267 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:31.232183    4470 patch_prober.go:29] interesting pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 07:33:31.232267 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:31.232232    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 07:33:32.008803 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.008777    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:32.117585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.117508    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-openshift-ca\") pod \"962feb66-3458-4d86-a7f7-690c4192c916\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") "
Nov 03 07:33:32.117752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.117617    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/962feb66-3458-4d86-a7f7-690c4192c916-kube-api-access-db4hf\" (UniqueName: \"kubernetes.io/projected/962feb66-3458-4d86-a7f7-690c4192c916-kube-api-access-db4hf\") pod \"962feb66-3458-4d86-a7f7-690c4192c916\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") "
Nov 03 07:33:32.117752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.117672    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-xks-ca\") pod \"962feb66-3458-4d86-a7f7-690c4192c916\" (UID: \"962feb66-3458-4d86-a7f7-690c4192c916\") "
Nov 03 07:33:32.117908 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.117878    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-openshift-ca" pod "962feb66-3458-4d86-a7f7-690c4192c916" (UID: "962feb66-3458-4d86-a7f7-690c4192c916"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:33:32.118410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.118305    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:33:32.119963 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.119941    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962feb66-3458-4d86-a7f7-690c4192c916-kube-api-access-db4hf" pod "962feb66-3458-4d86-a7f7-690c4192c916" (UID: "962feb66-3458-4d86-a7f7-690c4192c916"). InnerVolumeSpecName "kube-api-access-db4hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:33:32.120060 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.119966    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-xks-ca" pod "962feb66-3458-4d86-a7f7-690c4192c916" (UID: "962feb66-3458-4d86-a7f7-690c4192c916"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:33:32.219960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.219928    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db4hf\" (UniqueName: \"kubernetes.io/projected/962feb66-3458-4d86-a7f7-690c4192c916-kube-api-access-db4hf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:33:32.219960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.219959    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962feb66-3458-4d86-a7f7-690c4192c916-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:33:32.236871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.236842    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:33:32.237359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.237341    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:33:32.237943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.237923    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:33:32.858099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.858070    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"
Nov 03 07:33:32.858099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.858084    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn" event={"ID":"962feb66-3458-4d86-a7f7-690c4192c916","Type":"ContainerDied","Data":"34ee81ded5e320f7fb037dcd7d131ea317018fbf258ca5cc4a409797f68b7892"}
Nov 03 07:33:32.858521 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:33:32.858173    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34ee81ded5e320f7fb037dcd7d131ea317018fbf258ca5cc4a409797f68b7892"
Nov 03 07:33:34.760967 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:33:34.760932    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:34:02.511719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.511680    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"]
Nov 03 07:34:02.512819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.512796    4470 cpu_manager.go:414] "containerMap: removing container" podUID="962feb66-3458-4d86-a7f7-690c4192c916" containerName="revoke-keys"
Nov 03 07:34:02.512819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.512818    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="962feb66-3458-4d86-a7f7-690c4192c916" containerName="revoke-keys"
Nov 03 07:34:02.513010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.512956    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="962feb66-3458-4d86-a7f7-690c4192c916" containerName="revoke-keys"
Nov 03 07:34:02.516608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.516589    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:02.574603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.574557    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"]
Nov 03 07:34:02.592739 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.592702    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:02.592933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.592911    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:02.593281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.593256    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xjp\" (UniqueName: \"kubernetes.io/projected/d65e6300-b9e6-48cd-ac67-97f93c1d350c-kube-api-access-d9xjp\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:02.610169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.610124    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:34:02.619926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.619907    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:34:02.620039 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.619925    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:34:02.620039 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.619925    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:34:02.695211 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.695174    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:02.695402 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.695371    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xjp\" (UniqueName: \"kubernetes.io/projected/d65e6300-b9e6-48cd-ac67-97f93c1d350c-kube-api-access-d9xjp\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:02.695481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.695421    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:02.695783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.695763    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:02.697945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.697926    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:02.758883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.758848    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xjp\" (UniqueName: \"kubernetes.io/projected/d65e6300-b9e6-48cd-ac67-97f93c1d350c-kube-api-access-d9xjp\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:02.831858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:02.831821    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:03.221056 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:34:03.221024    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65e6300_b9e6_48cd_ac67_97f93c1d350c.slice/crio-fa333a17dc42d4fa0efebf3955ae79432cf85d5bcdafe990d7aeba1959441dac WatchSource:0}: Error finding container fa333a17dc42d4fa0efebf3955ae79432cf85d5bcdafe990d7aeba1959441dac: Status 404 returned error can't find the container with id fa333a17dc42d4fa0efebf3955ae79432cf85d5bcdafe990d7aeba1959441dac
Nov 03 07:34:03.221394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:03.221372    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"]
Nov 03 07:34:04.231178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:04.228888    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7" event={"ID":"d65e6300-b9e6-48cd-ac67-97f93c1d350c","Type":"ContainerStarted","Data":"0ee636498d12e3bf8e1b8a4625619e73030d85e662028c89203519fb837762bc"}
Nov 03 07:34:04.231178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:04.228952    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7" event={"ID":"d65e6300-b9e6-48cd-ac67-97f93c1d350c","Type":"ContainerStarted","Data":"fa333a17dc42d4fa0efebf3955ae79432cf85d5bcdafe990d7aeba1959441dac"}
Nov 03 07:34:04.764399 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:34:04.764369    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:34:05.253854 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.253765    4470 generic.go:356] "Generic (PLEG): container finished" podID="d65e6300-b9e6-48cd-ac67-97f93c1d350c" containerID="0ee636498d12e3bf8e1b8a4625619e73030d85e662028c89203519fb837762bc" exitCode=0
Nov 03 07:34:05.253854 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.253816    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7" event={"ID":"d65e6300-b9e6-48cd-ac67-97f93c1d350c","Type":"ContainerDied","Data":"0ee636498d12e3bf8e1b8a4625619e73030d85e662028c89203519fb837762bc"}
Nov 03 07:34:05.432543 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.432516    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:05.564512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.564480    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65e6300-b9e6-48cd-ac67-97f93c1d350c-kube-api-access-d9xjp\" (UniqueName: \"kubernetes.io/projected/d65e6300-b9e6-48cd-ac67-97f93c1d350c-kube-api-access-d9xjp\") pod \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") "
Nov 03 07:34:05.564733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.564693    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-xks-ca\") pod \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") "
Nov 03 07:34:05.564818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.564733    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-openshift-ca\") pod \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\" (UID: \"d65e6300-b9e6-48cd-ac67-97f93c1d350c\") "
Nov 03 07:34:05.565281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.565254    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-openshift-ca" pod "d65e6300-b9e6-48cd-ac67-97f93c1d350c" (UID: "d65e6300-b9e6-48cd-ac67-97f93c1d350c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:34:05.565529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.565497    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:05.570773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.570296    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65e6300-b9e6-48cd-ac67-97f93c1d350c-kube-api-access-d9xjp" pod "d65e6300-b9e6-48cd-ac67-97f93c1d350c" (UID: "d65e6300-b9e6-48cd-ac67-97f93c1d350c"). InnerVolumeSpecName "kube-api-access-d9xjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:05.570773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.570286    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-xks-ca" pod "d65e6300-b9e6-48cd-ac67-97f93c1d350c" (UID: "d65e6300-b9e6-48cd-ac67-97f93c1d350c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:34:05.666531 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.666496    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9xjp\" (UniqueName: \"kubernetes.io/projected/d65e6300-b9e6-48cd-ac67-97f93c1d350c-kube-api-access-d9xjp\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:05.666531 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:05.666531    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65e6300-b9e6-48cd-ac67-97f93c1d350c-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:06.262525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:06.262440    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"
Nov 03 07:34:06.262525 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:06.262489    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7" event={"ID":"d65e6300-b9e6-48cd-ac67-97f93c1d350c","Type":"ContainerDied","Data":"fa333a17dc42d4fa0efebf3955ae79432cf85d5bcdafe990d7aeba1959441dac"}
Nov 03 07:34:06.262952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:06.262534    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa333a17dc42d4fa0efebf3955ae79432cf85d5bcdafe990d7aeba1959441dac"
Nov 03 07:34:13.679958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.679924    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"]
Nov 03 07:34:13.680693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.680677    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d65e6300-b9e6-48cd-ac67-97f93c1d350c" containerName="revoke-keys"
Nov 03 07:34:13.680738 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.680694    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65e6300-b9e6-48cd-ac67-97f93c1d350c" containerName="revoke-keys"
Nov 03 07:34:13.680851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.680842    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65e6300-b9e6-48cd-ac67-97f93c1d350c" containerName="revoke-keys"
Nov 03 07:34:13.684736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.684722    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:13.742577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.742543    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"]
Nov 03 07:34:13.787819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.787786    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:34:13.787995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.787841    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:34:13.792648 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.792618    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:13.792829 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.792814    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4w28\" (UniqueName: \"kubernetes.io/projected/2d4c2854-383d-42a7-8724-7796e4ad5a0c-kube-api-access-n4w28\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:13.792992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.792974    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:13.807662 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.807632    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:34:13.838414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.838395    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:34:13.894010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.893979    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:13.894185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.894172    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4w28\" (UniqueName: \"kubernetes.io/projected/2d4c2854-383d-42a7-8724-7796e4ad5a0c-kube-api-access-n4w28\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:13.894375 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.894351    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:13.894628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.894611    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:13.896858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.896835    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:13.909718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.909687    4470 scope.go:122] "RemoveContainer" containerID="01d7226703108514cb6045f9c2a797e1efbd7640c80b0684e133d8080e2a8d46"
Nov 03 07:34:13.920413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.920396    4470 scope.go:122] "RemoveContainer" containerID="e4b8f84676844a9259acb6aab1cf15b68f71a4e9bdcc327ffa9eca82b504d965"
Nov 03 07:34:13.958322 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.958299    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4w28\" (UniqueName: \"kubernetes.io/projected/2d4c2854-383d-42a7-8724-7796e4ad5a0c-kube-api-access-n4w28\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:13.998751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:13.998730    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:14.338392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:14.338357    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"]
Nov 03 07:34:14.338698 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:14.338615    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerName="istio-proxy" containerID="cri-o://d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b" gracePeriod=30
Nov 03 07:34:14.345203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:14.345176    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"]
Nov 03 07:34:14.345436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:14.345404    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" podUID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" containerName="istio-proxy" containerID="cri-o://22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1" gracePeriod=30
Nov 03 07:34:14.384721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:14.384690    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"]
Nov 03 07:34:14.427111 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:34:14.427076    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4c2854_383d_42a7_8724_7796e4ad5a0c.slice/crio-afa60fa65e3d5e23b1d6d417e546009ac932fb06f7b8b63f5821f5ad11df725a WatchSource:0}: Error finding container afa60fa65e3d5e23b1d6d417e546009ac932fb06f7b8b63f5821f5ad11df725a: Status 404 returned error can't find the container with id afa60fa65e3d5e23b1d6d417e546009ac932fb06f7b8b63f5821f5ad11df725a
Nov 03 07:34:14.601706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:14.601624    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:34:14.601706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:14.601686    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" podUID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:34:15.262090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:15.262002    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"]
Nov 03 07:34:15.339416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:15.339378    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs" event={"ID":"2d4c2854-383d-42a7-8724-7796e4ad5a0c","Type":"ContainerStarted","Data":"1997a4525f9fc8b19f8136f5c967f614893a386597bbf1a8bab7d79d052e32ff"}
Nov 03 07:34:15.339595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:15.339434    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs" event={"ID":"2d4c2854-383d-42a7-8724-7796e4ad5a0c","Type":"ContainerStarted","Data":"afa60fa65e3d5e23b1d6d417e546009ac932fb06f7b8b63f5821f5ad11df725a"}
Nov 03 07:34:15.504993 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:15.504935    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs" podStartSLOduration=2.504920452 podStartE2EDuration="2.504920452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:34:15.503294515 +0000 UTC m=+2107.835663509" watchObservedRunningTime="2026-11-03 07:34:15.504920452 +0000 UTC m=+2107.837289445"
Nov 03 07:34:16.353741 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:16.353705    4470 generic.go:356] "Generic (PLEG): container finished" podID="2d4c2854-383d-42a7-8724-7796e4ad5a0c" containerID="1997a4525f9fc8b19f8136f5c967f614893a386597bbf1a8bab7d79d052e32ff" exitCode=0
Nov 03 07:34:16.354332 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:16.353791    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs" event={"ID":"2d4c2854-383d-42a7-8724-7796e4ad5a0c","Type":"ContainerDied","Data":"1997a4525f9fc8b19f8136f5c967f614893a386597bbf1a8bab7d79d052e32ff"}
Nov 03 07:34:17.531171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.531127    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:17.566369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.566328    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-openshift-ca\") pod \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") "
Nov 03 07:34:17.566550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.566403    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-xks-ca\") pod \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") "
Nov 03 07:34:17.566550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.566491    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4c2854-383d-42a7-8724-7796e4ad5a0c-kube-api-access-n4w28\" (UniqueName: \"kubernetes.io/projected/2d4c2854-383d-42a7-8724-7796e4ad5a0c-kube-api-access-n4w28\") pod \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\" (UID: \"2d4c2854-383d-42a7-8724-7796e4ad5a0c\") "
Nov 03 07:34:17.566805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.566778    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-openshift-ca" pod "2d4c2854-383d-42a7-8724-7796e4ad5a0c" (UID: "2d4c2854-383d-42a7-8724-7796e4ad5a0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:34:17.567010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.566991    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:17.568895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.568863    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4c2854-383d-42a7-8724-7796e4ad5a0c-kube-api-access-n4w28" pod "2d4c2854-383d-42a7-8724-7796e4ad5a0c" (UID: "2d4c2854-383d-42a7-8724-7796e4ad5a0c"). InnerVolumeSpecName "kube-api-access-n4w28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:17.568895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.568888    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-xks-ca" pod "2d4c2854-383d-42a7-8724-7796e4ad5a0c" (UID: "2d4c2854-383d-42a7-8724-7796e4ad5a0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:34:17.650505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.650420    4470 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:34:17.650505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.650475    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:34:17.668560 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.668523    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d4c2854-383d-42a7-8724-7796e4ad5a0c-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:17.668560 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:17.668565    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4w28\" (UniqueName: \"kubernetes.io/projected/2d4c2854-383d-42a7-8724-7796e4ad5a0c-kube-api-access-n4w28\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:18.374388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.374344    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs" event={"ID":"2d4c2854-383d-42a7-8724-7796e4ad5a0c","Type":"ContainerDied","Data":"afa60fa65e3d5e23b1d6d417e546009ac932fb06f7b8b63f5821f5ad11df725a"}
Nov 03 07:34:18.374598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.374453    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afa60fa65e3d5e23b1d6d417e546009ac932fb06f7b8b63f5821f5ad11df725a"
Nov 03 07:34:18.374598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.374538    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"
Nov 03 07:34:18.633913 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.633806    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"]
Nov 03 07:34:18.634585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.634569    4470 cpu_manager.go:414] "containerMap: removing container" podUID="2d4c2854-383d-42a7-8724-7796e4ad5a0c" containerName="revoke-keys"
Nov 03 07:34:18.634640 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.634587    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4c2854-383d-42a7-8724-7796e4ad5a0c" containerName="revoke-keys"
Nov 03 07:34:18.634711 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.634701    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4c2854-383d-42a7-8724-7796e4ad5a0c" containerName="revoke-keys"
Nov 03 07:34:18.638095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.638080    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:18.684118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.684087    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7q2\" (UniqueName: \"kubernetes.io/projected/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-kube-api-access-js7q2\") pod \"maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:18.684356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.684220    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:18.684356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.684312    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:18.693466 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.693436    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:34:18.696822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.696796    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"]
Nov 03 07:34:18.728068 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.728031    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:34:18.728506 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.728425    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:34:18.728506 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.728472    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:34:18.786240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.786208    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7q2\" (UniqueName: \"kubernetes.io/projected/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-kube-api-access-js7q2\") pod \"maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:18.786426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.786293    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:18.786426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.786363    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:18.787121 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.787102    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:18.788796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.788775    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:18.850678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.850642    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7q2\" (UniqueName: \"kubernetes.io/projected/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-kube-api-access-js7q2\") pod \"maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:18.952705 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:18.952623    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:19.336498 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:34:19.336465    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31049885_e9d4_41a0_8b46_c5caa7a0e2c8.slice/crio-c7008d9afad65351ecace7da4cf347c36ee8fbd8e8d33c4c41a668d8611d3cce WatchSource:0}: Error finding container c7008d9afad65351ecace7da4cf347c36ee8fbd8e8d33c4c41a668d8611d3cce: Status 404 returned error can't find the container with id c7008d9afad65351ecace7da4cf347c36ee8fbd8e8d33c4c41a668d8611d3cce
Nov 03 07:34:19.337060 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.337032    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"]
Nov 03 07:34:19.384343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.384312    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2" event={"ID":"31049885-e9d4-41a0-8b46-c5caa7a0e2c8","Type":"ContainerStarted","Data":"c7008d9afad65351ecace7da4cf347c36ee8fbd8e8d33c4c41a668d8611d3cce"}
Nov 03 07:34:19.662588 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.662563    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:34:19.685053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.685029    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:34:19.698542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.698510    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-data\") pod \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") "
Nov 03 07:34:19.698707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.698557    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istiod-ca-cert\") pod \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") "
Nov 03 07:34:19.698707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.698608    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-data\") pod \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") "
Nov 03 07:34:19.698707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.698631    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-podinfo\") pod \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") "
Nov 03 07:34:19.698707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.698675    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-certs\") pod \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") "
Nov 03 07:34:19.698906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.698726    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-token\" (UniqueName: \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-token\") pod \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") "
Nov 03 07:34:19.698906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.698782    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-credential-socket\") pod \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") "
Nov 03 07:34:19.698906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.698863    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-data" pod "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" (UID: "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:34:19.698906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.698882    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-socket\") pod \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") "
Nov 03 07:34:19.698906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.698877    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-data" pod "7011aff5-3fda-4fd7-ac52-ec1dc99205e8" (UID: "7011aff5-3fda-4fd7-ac52-ec1dc99205e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:34:19.699179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.698971    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istiod-ca-cert\") pod \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") "
Nov 03 07:34:19.699179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699004    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-podinfo\") pod \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") "
Nov 03 07:34:19.699179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699040    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-kube-api-access-rflmk\" (UniqueName: \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-kube-api-access-rflmk\") pod \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") "
Nov 03 07:34:19.699179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699074    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-certs\") pod \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") "
Nov 03 07:34:19.699179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699111    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-credential-socket" pod "7011aff5-3fda-4fd7-ac52-ec1dc99205e8" (UID: "7011aff5-3fda-4fd7-ac52-ec1dc99205e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:34:19.699179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699109    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-certs" pod "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" (UID: "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:34:19.699179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699156    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-socket" pod "7011aff5-3fda-4fd7-ac52-ec1dc99205e8" (UID: "7011aff5-3fda-4fd7-ac52-ec1dc99205e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:34:19.699179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699171    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istiod-ca-cert" pod "7011aff5-3fda-4fd7-ac52-ec1dc99205e8" (UID: "7011aff5-3fda-4fd7-ac52-ec1dc99205e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:34:19.699770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699173    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-envoy\") pod \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") "
Nov 03 07:34:19.699770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699365    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-envoy\") pod \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\" (UID: \"7011aff5-3fda-4fd7-ac52-ec1dc99205e8\") "
Nov 03 07:34:19.699770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699377    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-certs" pod "7011aff5-3fda-4fd7-ac52-ec1dc99205e8" (UID: "7011aff5-3fda-4fd7-ac52-ec1dc99205e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:34:19.699770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699405    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istiod-ca-cert" pod "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" (UID: "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:34:19.699770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699419    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-socket\") pod \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") "
Nov 03 07:34:19.699770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699468    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-credential-socket\") pod \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") "
Nov 03 07:34:19.699770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699506    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-kube-api-access-p6sb9\" (UniqueName: \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-kube-api-access-p6sb9\") pod \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") "
Nov 03 07:34:19.699770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699596    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-token\" (UniqueName: \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-token\") pod \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\" (UID: \"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb\") "
Nov 03 07:34:19.699770 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699635    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-socket" pod "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" (UID: "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:34:19.700388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.699857    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-credential-socket" pod "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" (UID: "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:34:19.700606 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.700568    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.700606 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.700596    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.700752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.700611    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.700752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.700625    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.700752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.700639    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.700752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.700655    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.700752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.700667    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.700752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.700680    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.700752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.700711    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.700752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.700726    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.701599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.701576    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-envoy" pod "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" (UID: "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:34:19.702003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.701938    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-token" pod "7011aff5-3fda-4fd7-ac52-ec1dc99205e8" (UID: "7011aff5-3fda-4fd7-ac52-ec1dc99205e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:19.702163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.702032    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-podinfo" pod "7011aff5-3fda-4fd7-ac52-ec1dc99205e8" (UID: "7011aff5-3fda-4fd7-ac52-ec1dc99205e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:34:19.702231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.702173    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-envoy" pod "7011aff5-3fda-4fd7-ac52-ec1dc99205e8" (UID: "7011aff5-3fda-4fd7-ac52-ec1dc99205e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:34:19.702578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.702555    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-kube-api-access-p6sb9" pod "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" (UID: "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb"). InnerVolumeSpecName "kube-api-access-p6sb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:19.702739 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.702718    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-podinfo" pod "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" (UID: "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:34:19.703186 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.703161    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-kube-api-access-rflmk" pod "7011aff5-3fda-4fd7-ac52-ec1dc99205e8" (UID: "7011aff5-3fda-4fd7-ac52-ec1dc99205e8"). InnerVolumeSpecName "kube-api-access-rflmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:19.703618 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.703598    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-token" pod "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" (UID: "a92a8b3b-a5fb-48c6-945e-8e09b712cbdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:19.802886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.802845    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6sb9\" (UniqueName: \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-kube-api-access-p6sb9\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.802886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.802888    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.803086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.802903    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.803086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.802916    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.803086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.802929    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.803086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.802944    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rflmk\" (UniqueName: \"kubernetes.io/projected/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-kube-api-access-rflmk\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.803086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.802958    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:19.803086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:19.802974    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7011aff5-3fda-4fd7-ac52-ec1dc99205e8-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:20.263084 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.262986    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr" podUID="e2905b11-1697-45de-8261-ff449c0cb075" containerName="payload-processing" containerID="cri-o://ea9a769f129d4cb852a739c75493e28780bce4e8eeaea4995dbb09f19fa6bce3" gracePeriod=25
Nov 03 07:34:20.396424 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.396390    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2" event={"ID":"31049885-e9d4-41a0-8b46-c5caa7a0e2c8","Type":"ContainerStarted","Data":"e9376f10aa463e04a908122af72b574cef85e614a07fc007f911cccca68e147c"}
Nov 03 07:34:20.398251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.398226    4470 generic.go:356] "Generic (PLEG): container finished" podID="e2905b11-1697-45de-8261-ff449c0cb075" containerID="ea9a769f129d4cb852a739c75493e28780bce4e8eeaea4995dbb09f19fa6bce3" exitCode=0
Nov 03 07:34:20.398366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.398300    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr" event={"ID":"e2905b11-1697-45de-8261-ff449c0cb075","Type":"ContainerDied","Data":"ea9a769f129d4cb852a739c75493e28780bce4e8eeaea4995dbb09f19fa6bce3"}
Nov 03 07:34:20.399749 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.399722    4470 generic.go:356] "Generic (PLEG): container finished" podID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerID="d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b" exitCode=0
Nov 03 07:34:20.399853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.399750    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" event={"ID":"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb","Type":"ContainerDied","Data":"d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b"}
Nov 03 07:34:20.399853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.399780    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"
Nov 03 07:34:20.399853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.399796    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx" event={"ID":"a92a8b3b-a5fb-48c6-945e-8e09b712cbdb","Type":"ContainerDied","Data":"2d61055a53a0eff278789e31f38b2495ff502810e43cb24b19b6cd49aed99502"}
Nov 03 07:34:20.399853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.399821    4470 scope.go:122] "RemoveContainer" containerID="d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b"
Nov 03 07:34:20.401273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.401251    4470 generic.go:356] "Generic (PLEG): container finished" podID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" containerID="22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1" exitCode=0
Nov 03 07:34:20.401391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.401293    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"
Nov 03 07:34:20.401391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.401330    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" event={"ID":"7011aff5-3fda-4fd7-ac52-ec1dc99205e8","Type":"ContainerDied","Data":"22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1"}
Nov 03 07:34:20.401391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.401374    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6" event={"ID":"7011aff5-3fda-4fd7-ac52-ec1dc99205e8","Type":"ContainerDied","Data":"7bec4985f9a97bc786e772be8b90a59ea9470fd6da8179e750ec314c6b1355a1"}
Nov 03 07:34:20.411614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.411597    4470 scope.go:122] "RemoveContainer" containerID="d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b"
Nov 03 07:34:20.411943 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:34:20.411924    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b\": container with ID starting with d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b not found: ID does not exist" containerID="d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b"
Nov 03 07:34:20.412000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.411954    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b"} err="failed to get container status \"d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b\": rpc error: code = NotFound desc = could not find container \"d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b\": container with ID starting with d962f5695052a175f9a44f47abbf89ed8fa655bcd1339b098877a9a81b96209b not found: ID does not exist"
Nov 03 07:34:20.412000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.411974    4470 scope.go:122] "RemoveContainer" containerID="22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1"
Nov 03 07:34:20.424183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.424118    4470 scope.go:122] "RemoveContainer" containerID="22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1"
Nov 03 07:34:20.424542 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:34:20.424519    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1\": container with ID starting with 22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1 not found: ID does not exist" containerID="22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1"
Nov 03 07:34:20.424613 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.424555    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1"} err="failed to get container status \"22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1\": rpc error: code = NotFound desc = could not find container \"22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1\": container with ID starting with 22805cf9cd381392c351527bf4a588824ec5e180db8af3860e6ea668f197f7b1 not found: ID does not exist"
Nov 03 07:34:20.522008 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.521909    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2" podStartSLOduration=2.521894196 podStartE2EDuration="2.521894196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:34:20.519277067 +0000 UTC m=+2112.851646065" watchObservedRunningTime="2026-11-03 07:34:20.521894196 +0000 UTC m=+2112.854263190"
Nov 03 07:34:20.560433 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.560410    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:34:20.617635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.617604    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2905b11-1697-45de-8261-ff449c0cb075-kube-api-access-kw2g5\" (UniqueName: \"kubernetes.io/projected/e2905b11-1697-45de-8261-ff449c0cb075-kube-api-access-kw2g5\") pod \"e2905b11-1697-45de-8261-ff449c0cb075\" (UID: \"e2905b11-1697-45de-8261-ff449c0cb075\") "
Nov 03 07:34:20.617837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.617822    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2905b11-1697-45de-8261-ff449c0cb075-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2905b11-1697-45de-8261-ff449c0cb075-plugins-config-volume\") pod \"e2905b11-1697-45de-8261-ff449c0cb075\" (UID: \"e2905b11-1697-45de-8261-ff449c0cb075\") "
Nov 03 07:34:20.618257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.618231    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2905b11-1697-45de-8261-ff449c0cb075-plugins-config-volume" pod "e2905b11-1697-45de-8261-ff449c0cb075" (UID: "e2905b11-1697-45de-8261-ff449c0cb075"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:34:20.618516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.618489    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2905b11-1697-45de-8261-ff449c0cb075-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:20.620327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.620305    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2905b11-1697-45de-8261-ff449c0cb075-kube-api-access-kw2g5" pod "e2905b11-1697-45de-8261-ff449c0cb075" (UID: "e2905b11-1697-45de-8261-ff449c0cb075"). InnerVolumeSpecName "kube-api-access-kw2g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:20.638643 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.638620    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"]
Nov 03 07:34:20.658161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.646103    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx"]
Nov 03 07:34:20.720092 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.720059    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw2g5\" (UniqueName: \"kubernetes.io/projected/e2905b11-1697-45de-8261-ff449c0cb075-kube-api-access-kw2g5\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:20.762798 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.762764    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"]
Nov 03 07:34:20.766934 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:20.766910    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6"]
Nov 03 07:34:21.412105 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:21.412070    4470 generic.go:356] "Generic (PLEG): container finished" podID="31049885-e9d4-41a0-8b46-c5caa7a0e2c8" containerID="e9376f10aa463e04a908122af72b574cef85e614a07fc007f911cccca68e147c" exitCode=0
Nov 03 07:34:21.412324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:21.412120    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2" event={"ID":"31049885-e9d4-41a0-8b46-c5caa7a0e2c8","Type":"ContainerDied","Data":"e9376f10aa463e04a908122af72b574cef85e614a07fc007f911cccca68e147c"}
Nov 03 07:34:21.413613 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:21.413580    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr" event={"ID":"e2905b11-1697-45de-8261-ff449c0cb075","Type":"ContainerDied","Data":"3a5600ab41fa8acaa464681b0ac45343036d0c2f698f1cbc356177f37d516858"}
Nov 03 07:34:21.413734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:21.413645    4470 scope.go:122] "RemoveContainer" containerID="ea9a769f129d4cb852a739c75493e28780bce4e8eeaea4995dbb09f19fa6bce3"
Nov 03 07:34:21.413734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:21.413592    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"
Nov 03 07:34:21.655718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:21.655683    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"]
Nov 03 07:34:21.661530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:21.661501    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr"]
Nov 03 07:34:21.809007 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:21.808927    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" path="/var/lib/kubelet/pods/7011aff5-3fda-4fd7-ac52-ec1dc99205e8/volumes"
Nov 03 07:34:21.809560 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:21.809543    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" path="/var/lib/kubelet/pods/a92a8b3b-a5fb-48c6-945e-8e09b712cbdb/volumes"
Nov 03 07:34:21.809990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:21.809974    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2905b11-1697-45de-8261-ff449c0cb075" path="/var/lib/kubelet/pods/e2905b11-1697-45de-8261-ff449c0cb075/volumes"
Nov 03 07:34:22.601977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.601949    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:22.622621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.622594    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"]
Nov 03 07:34:22.623354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623338    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" containerName="istio-proxy"
Nov 03 07:34:22.623409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623356    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" containerName="istio-proxy"
Nov 03 07:34:22.623409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623369    4470 cpu_manager.go:414] "containerMap: removing container" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerName="istio-proxy"
Nov 03 07:34:22.623409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623373    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerName="istio-proxy"
Nov 03 07:34:22.623409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623389    4470 cpu_manager.go:414] "containerMap: removing container" podUID="31049885-e9d4-41a0-8b46-c5caa7a0e2c8" containerName="revoke-keys"
Nov 03 07:34:22.623409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623395    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31049885-e9d4-41a0-8b46-c5caa7a0e2c8" containerName="revoke-keys"
Nov 03 07:34:22.623564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623416    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e2905b11-1697-45de-8261-ff449c0cb075" containerName="payload-processing"
Nov 03 07:34:22.623564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623421    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2905b11-1697-45de-8261-ff449c0cb075" containerName="payload-processing"
Nov 03 07:34:22.623564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623524    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7011aff5-3fda-4fd7-ac52-ec1dc99205e8" containerName="istio-proxy"
Nov 03 07:34:22.623564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623533    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92a8b3b-a5fb-48c6-945e-8e09b712cbdb" containerName="istio-proxy"
Nov 03 07:34:22.623564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623542    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="31049885-e9d4-41a0-8b46-c5caa7a0e2c8" containerName="revoke-keys"
Nov 03 07:34:22.623564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.623550    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2905b11-1697-45de-8261-ff449c0cb075" containerName="payload-processing"
Nov 03 07:34:22.626983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.626968    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:22.651903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.651873    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-kube-api-access-js7q2\" (UniqueName: \"kubernetes.io/projected/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-kube-api-access-js7q2\") pod \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") "
Nov 03 07:34:22.652065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.651907    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-xks-ca\") pod \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") "
Nov 03 07:34:22.652065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.652005    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-openshift-ca\") pod \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\" (UID: \"31049885-e9d4-41a0-8b46-c5caa7a0e2c8\") "
Nov 03 07:34:22.652329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.652307    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:22.652447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.652355    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqrg6\" (UniqueName: \"kubernetes.io/projected/71eb7f69-9eed-45da-a5e3-d696cbe525cb-kube-api-access-cqrg6\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:22.652447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.652390    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-openshift-ca" pod "31049885-e9d4-41a0-8b46-c5caa7a0e2c8" (UID: "31049885-e9d4-41a0-8b46-c5caa7a0e2c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:34:22.652938 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.652915    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:22.653210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.653188    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:22.654262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.654242    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-xks-ca" pod "31049885-e9d4-41a0-8b46-c5caa7a0e2c8" (UID: "31049885-e9d4-41a0-8b46-c5caa7a0e2c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:34:22.654342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.654325    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-kube-api-access-js7q2" pod "31049885-e9d4-41a0-8b46-c5caa7a0e2c8" (UID: "31049885-e9d4-41a0-8b46-c5caa7a0e2c8"). InnerVolumeSpecName "kube-api-access-js7q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:22.686368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.686303    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"]
Nov 03 07:34:22.754664 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.754627    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:22.754861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.754750    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:22.754861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.754779    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqrg6\" (UniqueName: \"kubernetes.io/projected/71eb7f69-9eed-45da-a5e3-d696cbe525cb-kube-api-access-cqrg6\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:22.755222 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.755192    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js7q2\" (UniqueName: \"kubernetes.io/projected/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-kube-api-access-js7q2\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:22.755222 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.755219    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31049885-e9d4-41a0-8b46-c5caa7a0e2c8-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:22.755638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.755614    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:22.757405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.757359    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:22.819021 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.818991    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqrg6\" (UniqueName: \"kubernetes.io/projected/71eb7f69-9eed-45da-a5e3-d696cbe525cb-kube-api-access-cqrg6\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:22.940577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:22.940505    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:23.322355 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:34:23.322319    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71eb7f69_9eed_45da_a5e3_d696cbe525cb.slice/crio-c167c3549187c421913c6e569ad4299bf8476a7c22b5793869c1886266b7e6bf WatchSource:0}: Error finding container c167c3549187c421913c6e569ad4299bf8476a7c22b5793869c1886266b7e6bf: Status 404 returned error can't find the container with id c167c3549187c421913c6e569ad4299bf8476a7c22b5793869c1886266b7e6bf
Nov 03 07:34:23.322528 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:23.322508    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"]
Nov 03 07:34:23.439951 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:23.439914    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"
Nov 03 07:34:23.439951 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:23.439969    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2" event={"ID":"31049885-e9d4-41a0-8b46-c5caa7a0e2c8","Type":"ContainerDied","Data":"c7008d9afad65351ecace7da4cf347c36ee8fbd8e8d33c4c41a668d8611d3cce"}
Nov 03 07:34:23.440245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:23.440022    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7008d9afad65351ecace7da4cf347c36ee8fbd8e8d33c4c41a668d8611d3cce"
Nov 03 07:34:23.441698 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:23.441658    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq" event={"ID":"71eb7f69-9eed-45da-a5e3-d696cbe525cb","Type":"ContainerStarted","Data":"c167c3549187c421913c6e569ad4299bf8476a7c22b5793869c1886266b7e6bf"}
Nov 03 07:34:24.450880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:24.450845    4470 generic.go:356] "Generic (PLEG): container finished" podID="71eb7f69-9eed-45da-a5e3-d696cbe525cb" containerID="9d153c8ecd3dfcba6bb8baf4ea48e42dd4ea064f65b2e21a3b1f089c2479c08c" exitCode=0
Nov 03 07:34:24.451343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:24.450995    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq" event={"ID":"71eb7f69-9eed-45da-a5e3-d696cbe525cb","Type":"ContainerDied","Data":"9d153c8ecd3dfcba6bb8baf4ea48e42dd4ea064f65b2e21a3b1f089c2479c08c"}
Nov 03 07:34:25.035023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:25.034991    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"]
Nov 03 07:34:25.721201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:25.721165    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:25.806576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:25.806494    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71eb7f69-9eed-45da-a5e3-d696cbe525cb-kube-api-access-cqrg6\" (UniqueName: \"kubernetes.io/projected/71eb7f69-9eed-45da-a5e3-d696cbe525cb-kube-api-access-cqrg6\") pod \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") "
Nov 03 07:34:25.806764 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:25.806589    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-openshift-ca\") pod \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") "
Nov 03 07:34:25.806764 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:25.806649    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-xks-ca\") pod \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\" (UID: \"71eb7f69-9eed-45da-a5e3-d696cbe525cb\") "
Nov 03 07:34:25.806953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:25.806919    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-openshift-ca" pod "71eb7f69-9eed-45da-a5e3-d696cbe525cb" (UID: "71eb7f69-9eed-45da-a5e3-d696cbe525cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:34:25.807074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:25.807055    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:25.809149 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:25.809105    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-xks-ca" pod "71eb7f69-9eed-45da-a5e3-d696cbe525cb" (UID: "71eb7f69-9eed-45da-a5e3-d696cbe525cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:34:25.809413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:25.809384    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71eb7f69-9eed-45da-a5e3-d696cbe525cb-kube-api-access-cqrg6" pod "71eb7f69-9eed-45da-a5e3-d696cbe525cb" (UID: "71eb7f69-9eed-45da-a5e3-d696cbe525cb"). InnerVolumeSpecName "kube-api-access-cqrg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:25.908657 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:25.908618    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqrg6\" (UniqueName: \"kubernetes.io/projected/71eb7f69-9eed-45da-a5e3-d696cbe525cb-kube-api-access-cqrg6\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:25.908657 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:25.908656    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71eb7f69-9eed-45da-a5e3-d696cbe525cb-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:26.535160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:26.533829    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq" event={"ID":"71eb7f69-9eed-45da-a5e3-d696cbe525cb","Type":"ContainerDied","Data":"c167c3549187c421913c6e569ad4299bf8476a7c22b5793869c1886266b7e6bf"}
Nov 03 07:34:26.535160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:26.533890    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c167c3549187c421913c6e569ad4299bf8476a7c22b5793869c1886266b7e6bf"
Nov 03 07:34:26.535160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:26.533960    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"
Nov 03 07:34:29.949982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:29.949950    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"]
Nov 03 07:34:30.035560 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.035520    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd" podUID="7f3f0492-22c0-457d-8e64-2da01d8c1370" containerName="payload-processing" containerID="cri-o://42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131" gracePeriod=25
Nov 03 07:34:30.066970 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.066932    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"]
Nov 03 07:34:30.332422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.332397    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:34:30.491415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.491315    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3f0492-22c0-457d-8e64-2da01d8c1370-kube-api-access-wh8mz\" (UniqueName: \"kubernetes.io/projected/7f3f0492-22c0-457d-8e64-2da01d8c1370-kube-api-access-wh8mz\") pod \"7f3f0492-22c0-457d-8e64-2da01d8c1370\" (UID: \"7f3f0492-22c0-457d-8e64-2da01d8c1370\") "
Nov 03 07:34:30.491715 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.491695    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3f0492-22c0-457d-8e64-2da01d8c1370-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3f0492-22c0-457d-8e64-2da01d8c1370-plugins-config-volume\") pod \"7f3f0492-22c0-457d-8e64-2da01d8c1370\" (UID: \"7f3f0492-22c0-457d-8e64-2da01d8c1370\") "
Nov 03 07:34:30.492515 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.492491    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3f0492-22c0-457d-8e64-2da01d8c1370-plugins-config-volume" pod "7f3f0492-22c0-457d-8e64-2da01d8c1370" (UID: "7f3f0492-22c0-457d-8e64-2da01d8c1370"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:34:30.496186 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.496158    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3f0492-22c0-457d-8e64-2da01d8c1370-kube-api-access-wh8mz" pod "7f3f0492-22c0-457d-8e64-2da01d8c1370" (UID: "7f3f0492-22c0-457d-8e64-2da01d8c1370"). InnerVolumeSpecName "kube-api-access-wh8mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:30.570507 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.570476    4470 generic.go:356] "Generic (PLEG): container finished" podID="7f3f0492-22c0-457d-8e64-2da01d8c1370" containerID="42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131" exitCode=0
Nov 03 07:34:30.570683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.570523    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"
Nov 03 07:34:30.570683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.570566    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd" event={"ID":"7f3f0492-22c0-457d-8e64-2da01d8c1370","Type":"ContainerDied","Data":"42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131"}
Nov 03 07:34:30.570683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.570622    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd" event={"ID":"7f3f0492-22c0-457d-8e64-2da01d8c1370","Type":"ContainerDied","Data":"3e631d982971fef7f5f7899cdaa3d329726c40a095157acebecc23641a7940b6"}
Nov 03 07:34:30.570683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.570650    4470 scope.go:122] "RemoveContainer" containerID="42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131"
Nov 03 07:34:30.582062 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.582044    4470 scope.go:122] "RemoveContainer" containerID="42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131"
Nov 03 07:34:30.582352 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:34:30.582335    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131\": container with ID starting with 42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131 not found: ID does not exist" containerID="42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131"
Nov 03 07:34:30.582407 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.582360    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131"} err="failed to get container status \"42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131\": rpc error: code = NotFound desc = could not find container \"42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131\": container with ID starting with 42edf8c98b3606c79c1fba4dcf9de29622503bc130f45278cb85eb0ee4230131 not found: ID does not exist"
Nov 03 07:34:30.593679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.593655    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3f0492-22c0-457d-8e64-2da01d8c1370-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:30.593679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.593680    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh8mz\" (UniqueName: \"kubernetes.io/projected/7f3f0492-22c0-457d-8e64-2da01d8c1370-kube-api-access-wh8mz\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:30.745623 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.745531    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"]
Nov 03 07:34:30.751555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:30.751521    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd"]
Nov 03 07:34:31.882911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:31.882867    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3f0492-22c0-457d-8e64-2da01d8c1370" path="/var/lib/kubelet/pods/7f3f0492-22c0-457d-8e64-2da01d8c1370/volumes"
Nov 03 07:34:34.033252 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:34.033212    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"]
Nov 03 07:34:34.083699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:34.083656    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-418f41-g5ngs"]
Nov 03 07:34:34.761166 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:34:34.761103    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:34:34.950537 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:34.950479    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm" podUID="26d59cce-9220-4573-abb1-a17d6bedc1c9" containerName="payload-processing" containerID="cri-o://3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb" gracePeriod=25
Nov 03 07:34:35.067718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.067679    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs" podUID="d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495" containerName="payload-pre-processing" containerID="cri-o://4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9" gracePeriod=25
Nov 03 07:34:35.281169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.278774    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:34:35.358270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.355732    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d59cce-9220-4573-abb1-a17d6bedc1c9-kube-api-access-zwp6c\" (UniqueName: \"kubernetes.io/projected/26d59cce-9220-4573-abb1-a17d6bedc1c9-kube-api-access-zwp6c\") pod \"26d59cce-9220-4573-abb1-a17d6bedc1c9\" (UID: \"26d59cce-9220-4573-abb1-a17d6bedc1c9\") "
Nov 03 07:34:35.358270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.355899    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d59cce-9220-4573-abb1-a17d6bedc1c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d59cce-9220-4573-abb1-a17d6bedc1c9-plugins-config-volume\") pod \"26d59cce-9220-4573-abb1-a17d6bedc1c9\" (UID: \"26d59cce-9220-4573-abb1-a17d6bedc1c9\") "
Nov 03 07:34:35.375404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.359278    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d59cce-9220-4573-abb1-a17d6bedc1c9-plugins-config-volume" pod "26d59cce-9220-4573-abb1-a17d6bedc1c9" (UID: "26d59cce-9220-4573-abb1-a17d6bedc1c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:34:35.375404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.367914    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d59cce-9220-4573-abb1-a17d6bedc1c9-kube-api-access-zwp6c" pod "26d59cce-9220-4573-abb1-a17d6bedc1c9" (UID: "26d59cce-9220-4573-abb1-a17d6bedc1c9"). InnerVolumeSpecName "kube-api-access-zwp6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:35.461169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.452651    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:34:35.461169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.458727    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwp6c\" (UniqueName: \"kubernetes.io/projected/26d59cce-9220-4573-abb1-a17d6bedc1c9-kube-api-access-zwp6c\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:35.461169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.458783    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d59cce-9220-4573-abb1-a17d6bedc1c9-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:35.559443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.559410    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-plugins-config-volume\") pod \"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495\" (UID: \"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495\") "
Nov 03 07:34:35.559745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.559725    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-kube-api-access-9wl6r\" (UniqueName: \"kubernetes.io/projected/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-kube-api-access-9wl6r\") pod \"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495\" (UID: \"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495\") "
Nov 03 07:34:35.559841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.559767    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-plugins-config-volume" pod "d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495" (UID: "d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:34:35.560282 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.560263    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:35.561889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.561869    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-kube-api-access-9wl6r" pod "d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495" (UID: "d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495"). InnerVolumeSpecName "kube-api-access-9wl6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:34:35.661614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.661532    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wl6r\" (UniqueName: \"kubernetes.io/projected/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495-kube-api-access-9wl6r\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:34:35.665585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.665556    4470 generic.go:356] "Generic (PLEG): container finished" podID="d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495" containerID="4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9" exitCode=0
Nov 03 07:34:35.665715 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.665606    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"
Nov 03 07:34:35.665715 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.665632    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs" event={"ID":"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495","Type":"ContainerDied","Data":"4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9"}
Nov 03 07:34:35.665715 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.665682    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs" event={"ID":"d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495","Type":"ContainerDied","Data":"148013a8a4557c066d236d1b91f97af33908846e2e508006a1fda0948a2b9e3b"}
Nov 03 07:34:35.665715 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.665706    4470 scope.go:122] "RemoveContainer" containerID="4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9"
Nov 03 07:34:35.667253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.667231    4470 generic.go:356] "Generic (PLEG): container finished" podID="26d59cce-9220-4573-abb1-a17d6bedc1c9" containerID="3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb" exitCode=0
Nov 03 07:34:35.667342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.667276    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"
Nov 03 07:34:35.667342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.667306    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm" event={"ID":"26d59cce-9220-4573-abb1-a17d6bedc1c9","Type":"ContainerDied","Data":"3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb"}
Nov 03 07:34:35.667456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.667352    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm" event={"ID":"26d59cce-9220-4573-abb1-a17d6bedc1c9","Type":"ContainerDied","Data":"10fa226a822e15e084952ff94f4483963d9125770f39791d1cb907d968d50f10"}
Nov 03 07:34:35.679805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.679784    4470 scope.go:122] "RemoveContainer" containerID="4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9"
Nov 03 07:34:35.680077 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:34:35.680052    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9\": container with ID starting with 4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9 not found: ID does not exist" containerID="4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9"
Nov 03 07:34:35.680182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.680090    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9"} err="failed to get container status \"4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9\": rpc error: code = NotFound desc = could not find container \"4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9\": container with ID starting with 4e81864aa11809cb8ebe63f60e19a79b92c36e7c46b9213dfe9906065ea6aac9 not found: ID does not exist"
Nov 03 07:34:35.680182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.680115    4470 scope.go:122] "RemoveContainer" containerID="3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb"
Nov 03 07:34:35.690048 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.690033    4470 scope.go:122] "RemoveContainer" containerID="3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb"
Nov 03 07:34:35.690315 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:34:35.690297    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb\": container with ID starting with 3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb not found: ID does not exist" containerID="3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb"
Nov 03 07:34:35.690391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.690320    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb"} err="failed to get container status \"3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb\": rpc error: code = NotFound desc = could not find container \"3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb\": container with ID starting with 3582054fdc4431dd554a5dd39f5253e4e8c68e15514542c1c187b7e8794662cb not found: ID does not exist"
Nov 03 07:34:35.819285 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.819252    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ffac1f-d94c-4998-88b0-872da521e534" path="/var/lib/kubelet/pods/f4ffac1f-d94c-4998-88b0-872da521e534/volumes"
Nov 03 07:34:35.837749 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.837707    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"]
Nov 03 07:34:35.847568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.847543    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm"]
Nov 03 07:34:35.969965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.969532    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"]
Nov 03 07:34:35.973549 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:35.973511    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs"]
Nov 03 07:34:37.808555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:37.808522    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d59cce-9220-4573-abb1-a17d6bedc1c9" path="/var/lib/kubelet/pods/26d59cce-9220-4573-abb1-a17d6bedc1c9/volumes"
Nov 03 07:34:37.809099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:37.809079    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495" path="/var/lib/kubelet/pods/d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495/volumes"
Nov 03 07:34:41.056197 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:41.056148    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"]
Nov 03 07:34:41.105711 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:41.105676    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f56d732-h459g"]
Nov 03 07:34:41.904802 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:34:41.904696    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764a6883-b3a7-4efb-a411-9fd14e758a6a" path="/var/lib/kubelet/pods/764a6883-b3a7-4efb-a411-9fd14e758a6a/volumes"
Nov 03 07:35:04.760518 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:35:04.760481    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:35:07.221195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:07.221156    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"]
Nov 03 07:35:07.270543 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:07.270504    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-k7nl7"]
Nov 03 07:35:07.809450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:07.809421    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c7b2070-63f2-4492-aa27-feda9aecadd5" path="/var/lib/kubelet/pods/0c7b2070-63f2-4492-aa27-feda9aecadd5/volumes"
Nov 03 07:35:13.953667 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:13.953635    4470 scope.go:122] "RemoveContainer" containerID="a8d831105096541b36757ce07a7c773a9ae532f9e91382feb63d41a58a270629"
Nov 03 07:35:24.699489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:24.699451    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"]
Nov 03 07:35:24.749390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:24.749360    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5183ae-h8vp7"]
Nov 03 07:35:25.807510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:25.807481    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65e6300-b9e6-48cd-ac67-97f93c1d350c" path="/var/lib/kubelet/pods/d65e6300-b9e6-48cd-ac67-97f93c1d350c/volumes"
Nov 03 07:35:30.747239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:30.747174    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"]
Nov 03 07:35:30.797305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:30.797270    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-acdd08-84hfs"]
Nov 03 07:35:31.809214 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:31.809180    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4c2854-383d-42a7-8724-7796e4ad5a0c" path="/var/lib/kubelet/pods/2d4c2854-383d-42a7-8724-7796e4ad5a0c/volumes"
Nov 03 07:35:33.249995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:33.249962    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"]
Nov 03 07:35:33.250621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:33.250233    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy" containerID="cri-o://7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36" gracePeriod=30
Nov 03 07:35:34.122265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.122231    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"]
Nov 03 07:35:34.122915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.122902    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495" containerName="payload-pre-processing"
Nov 03 07:35:34.122965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.122917    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495" containerName="payload-pre-processing"
Nov 03 07:35:34.122965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.122940    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7f3f0492-22c0-457d-8e64-2da01d8c1370" containerName="payload-processing"
Nov 03 07:35:34.122965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.122945    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3f0492-22c0-457d-8e64-2da01d8c1370" containerName="payload-processing"
Nov 03 07:35:34.123065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.122971    4470 cpu_manager.go:414] "containerMap: removing container" podUID="26d59cce-9220-4573-abb1-a17d6bedc1c9" containerName="payload-processing"
Nov 03 07:35:34.123065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.122975    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d59cce-9220-4573-abb1-a17d6bedc1c9" containerName="payload-processing"
Nov 03 07:35:34.123065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.122986    4470 cpu_manager.go:414] "containerMap: removing container" podUID="71eb7f69-9eed-45da-a5e3-d696cbe525cb" containerName="revoke-keys"
Nov 03 07:35:34.123065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.122990    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71eb7f69-9eed-45da-a5e3-d696cbe525cb" containerName="revoke-keys"
Nov 03 07:35:34.123230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.123091    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d59cce-9220-4573-abb1-a17d6bedc1c9" containerName="payload-processing"
Nov 03 07:35:34.123230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.123101    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="71eb7f69-9eed-45da-a5e3-d696cbe525cb" containerName="revoke-keys"
Nov 03 07:35:34.123230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.123110    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a85e2c-44c2-4b66-9ae4-ee6da8ad4495" containerName="payload-pre-processing"
Nov 03 07:35:34.123230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.123118    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3f0492-22c0-457d-8e64-2da01d8c1370" containerName="payload-processing"
Nov 03 07:35:34.126897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.126881    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:34.187862 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.187831    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"]
Nov 03 07:35:34.195768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.195737    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqkw7\" (UniqueName: \"kubernetes.io/projected/b889dc0f-b0f6-4757-bc23-d983977bd559-kube-api-access-zqkw7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:34.195937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.195802    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:34.195937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.195887    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:34.233332 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.233289    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:35:34.233531 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.233379    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:35:34.280279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.280250    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:35:34.280720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.280449    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:35:34.297101 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.297076    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqkw7\" (UniqueName: \"kubernetes.io/projected/b889dc0f-b0f6-4757-bc23-d983977bd559-kube-api-access-zqkw7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:34.297254 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.297126    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:34.297254 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.297209    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:34.297920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.297900    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:34.299676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.299654    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:34.361406 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.361378    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqkw7\" (UniqueName: \"kubernetes.io/projected/b889dc0f-b0f6-4757-bc23-d983977bd559-kube-api-access-zqkw7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:34.441243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.441165    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:34.761584 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:35:34.761503    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:35:34.824506 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:34.824474    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"]
Nov 03 07:35:34.824741 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:35:34.824720    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb889dc0f_b0f6_4757_bc23_d983977bd559.slice/crio-4f7eb1167312b9e91629e5711e6daf5d0afa73da8f42ab0a3de52c78742edb3b WatchSource:0}: Error finding container 4f7eb1167312b9e91629e5711e6daf5d0afa73da8f42ab0a3de52c78742edb3b: Status 404 returned error can't find the container with id 4f7eb1167312b9e91629e5711e6daf5d0afa73da8f42ab0a3de52c78742edb3b
Nov 03 07:35:35.221234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:35.221195    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4" event={"ID":"b889dc0f-b0f6-4757-bc23-d983977bd559","Type":"ContainerStarted","Data":"2b1a62896a0782d1ef02a46126aec2fe83b4d880facad0a9f955ad700542d2f6"}
Nov 03 07:35:35.221434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:35.221263    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4" event={"ID":"b889dc0f-b0f6-4757-bc23-d983977bd559","Type":"ContainerStarted","Data":"4f7eb1167312b9e91629e5711e6daf5d0afa73da8f42ab0a3de52c78742edb3b"}
Nov 03 07:35:35.344756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:35.344658    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4" podStartSLOduration=1.344644535 podStartE2EDuration="1.344644535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:35:35.340816311 +0000 UTC m=+2187.673185306" watchObservedRunningTime="2026-11-03 07:35:35.344644535 +0000 UTC m=+2187.677013528"
Nov 03 07:35:36.231378 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:36.231346    4470 generic.go:356] "Generic (PLEG): container finished" podID="b889dc0f-b0f6-4757-bc23-d983977bd559" containerID="2b1a62896a0782d1ef02a46126aec2fe83b4d880facad0a9f955ad700542d2f6" exitCode=0
Nov 03 07:35:36.231561 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:36.231426    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4" event={"ID":"b889dc0f-b0f6-4757-bc23-d983977bd559","Type":"ContainerDied","Data":"2b1a62896a0782d1ef02a46126aec2fe83b4d880facad0a9f955ad700542d2f6"}
Nov 03 07:35:37.415922 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:37.415894    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:37.442992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:37.442952    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-openshift-ca\") pod \"b889dc0f-b0f6-4757-bc23-d983977bd559\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") "
Nov 03 07:35:37.443201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:37.443010    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-xks-ca\") pod \"b889dc0f-b0f6-4757-bc23-d983977bd559\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") "
Nov 03 07:35:37.443201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:37.443121    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b889dc0f-b0f6-4757-bc23-d983977bd559-kube-api-access-zqkw7\" (UniqueName: \"kubernetes.io/projected/b889dc0f-b0f6-4757-bc23-d983977bd559-kube-api-access-zqkw7\") pod \"b889dc0f-b0f6-4757-bc23-d983977bd559\" (UID: \"b889dc0f-b0f6-4757-bc23-d983977bd559\") "
Nov 03 07:35:37.443438 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:37.443410    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-openshift-ca" pod "b889dc0f-b0f6-4757-bc23-d983977bd559" (UID: "b889dc0f-b0f6-4757-bc23-d983977bd559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:35:37.443820 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:37.443804    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:37.445545 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:37.445518    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-xks-ca" pod "b889dc0f-b0f6-4757-bc23-d983977bd559" (UID: "b889dc0f-b0f6-4757-bc23-d983977bd559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:35:37.445658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:37.445571    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b889dc0f-b0f6-4757-bc23-d983977bd559-kube-api-access-zqkw7" pod "b889dc0f-b0f6-4757-bc23-d983977bd559" (UID: "b889dc0f-b0f6-4757-bc23-d983977bd559"). InnerVolumeSpecName "kube-api-access-zqkw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:35:37.545403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:37.545323    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b889dc0f-b0f6-4757-bc23-d983977bd559-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:37.545403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:37.545351    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqkw7\" (UniqueName: \"kubernetes.io/projected/b889dc0f-b0f6-4757-bc23-d983977bd559-kube-api-access-zqkw7\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:38.248240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.248161    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"
Nov 03 07:35:38.248240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.248181    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4" event={"ID":"b889dc0f-b0f6-4757-bc23-d983977bd559","Type":"ContainerDied","Data":"4f7eb1167312b9e91629e5711e6daf5d0afa73da8f42ab0a3de52c78742edb3b"}
Nov 03 07:35:38.248240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.248230    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f7eb1167312b9e91629e5711e6daf5d0afa73da8f42ab0a3de52c78742edb3b"
Nov 03 07:35:38.573298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.573273    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:35:38.659729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.659692    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-envoy\") pod \"eabd31f7-12a2-4753-8a37-bc835e84615d\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") "
Nov 03 07:35:38.659914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.659771    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-socket\") pod \"eabd31f7-12a2-4753-8a37-bc835e84615d\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") "
Nov 03 07:35:38.659914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.659811    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-podinfo\") pod \"eabd31f7-12a2-4753-8a37-bc835e84615d\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") "
Nov 03 07:35:38.659914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.659843    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabd31f7-12a2-4753-8a37-bc835e84615d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabd31f7-12a2-4753-8a37-bc835e84615d-istiod-ca-cert\") pod \"eabd31f7-12a2-4753-8a37-bc835e84615d\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") "
Nov 03 07:35:38.660066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.659915    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-kube-api-access-9nhdg\" (UniqueName: \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-kube-api-access-9nhdg\") pod \"eabd31f7-12a2-4753-8a37-bc835e84615d\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") "
Nov 03 07:35:38.660066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.659956    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-token\" (UniqueName: \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-token\") pod \"eabd31f7-12a2-4753-8a37-bc835e84615d\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") "
Nov 03 07:35:38.660066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.659986    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-certs\") pod \"eabd31f7-12a2-4753-8a37-bc835e84615d\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") "
Nov 03 07:35:38.660289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.660072    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-data\") pod \"eabd31f7-12a2-4753-8a37-bc835e84615d\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") "
Nov 03 07:35:38.660289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.660092    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-socket" pod "eabd31f7-12a2-4753-8a37-bc835e84615d" (UID: "eabd31f7-12a2-4753-8a37-bc835e84615d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:35:38.660419 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.660387    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-data" pod "eabd31f7-12a2-4753-8a37-bc835e84615d" (UID: "eabd31f7-12a2-4753-8a37-bc835e84615d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:35:38.660474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.660408    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-certs" pod "eabd31f7-12a2-4753-8a37-bc835e84615d" (UID: "eabd31f7-12a2-4753-8a37-bc835e84615d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:35:38.660474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.660433    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabd31f7-12a2-4753-8a37-bc835e84615d-istiod-ca-cert" pod "eabd31f7-12a2-4753-8a37-bc835e84615d" (UID: "eabd31f7-12a2-4753-8a37-bc835e84615d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:35:38.660712 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.660644    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-credential-socket\") pod \"eabd31f7-12a2-4753-8a37-bc835e84615d\" (UID: \"eabd31f7-12a2-4753-8a37-bc835e84615d\") "
Nov 03 07:35:38.661019 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.660993    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-credential-socket" pod "eabd31f7-12a2-4753-8a37-bc835e84615d" (UID: "eabd31f7-12a2-4753-8a37-bc835e84615d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:35:38.661516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.661497    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:38.661576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.661525    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:38.661576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.661542    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabd31f7-12a2-4753-8a37-bc835e84615d-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:38.661576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.661554    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:38.661576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.661569    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:38.662344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.662321    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-kube-api-access-9nhdg" pod "eabd31f7-12a2-4753-8a37-bc835e84615d" (UID: "eabd31f7-12a2-4753-8a37-bc835e84615d"). InnerVolumeSpecName "kube-api-access-9nhdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:35:38.662450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.662378    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-envoy" pod "eabd31f7-12a2-4753-8a37-bc835e84615d" (UID: "eabd31f7-12a2-4753-8a37-bc835e84615d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:35:38.662712 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.662696    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-token" pod "eabd31f7-12a2-4753-8a37-bc835e84615d" (UID: "eabd31f7-12a2-4753-8a37-bc835e84615d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:35:38.662777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.662730    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-podinfo" pod "eabd31f7-12a2-4753-8a37-bc835e84615d" (UID: "eabd31f7-12a2-4753-8a37-bc835e84615d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:35:38.763734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.763564    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:38.763734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.763616    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:38.763734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.763631    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nhdg\" (UniqueName: \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-kube-api-access-9nhdg\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:38.763734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:38.763646    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabd31f7-12a2-4753-8a37-bc835e84615d-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:39.257809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.257705    4470 generic.go:356] "Generic (PLEG): container finished" podID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerID="7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36" exitCode=0
Nov 03 07:35:39.258050 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.257837    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" event={"ID":"eabd31f7-12a2-4753-8a37-bc835e84615d","Type":"ContainerDied","Data":"7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36"}
Nov 03 07:35:39.258050 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.257976    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf" event={"ID":"eabd31f7-12a2-4753-8a37-bc835e84615d","Type":"ContainerDied","Data":"45ed25babd4bde5c17bc6e2e3c97a3b74a6bcc5b8a7edf45c9c38f5a630dc9ac"}
Nov 03 07:35:39.258050 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.258019    4470 scope.go:122] "RemoveContainer" containerID="7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36"
Nov 03 07:35:39.258249 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.257774    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"
Nov 03 07:35:39.270043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.270022    4470 scope.go:122] "RemoveContainer" containerID="35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d"
Nov 03 07:35:39.279923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.279903    4470 scope.go:122] "RemoveContainer" containerID="7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36"
Nov 03 07:35:39.280244 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:35:39.280225    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36\": container with ID starting with 7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36 not found: ID does not exist" containerID="7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36"
Nov 03 07:35:39.280304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.280253    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36"} err="failed to get container status \"7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36\": rpc error: code = NotFound desc = could not find container \"7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36\": container with ID starting with 7502c4af75aaf4425e51766110a7cf087958def4e20f3bf7c8a028b597ee9b36 not found: ID does not exist"
Nov 03 07:35:39.280304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.280269    4470 scope.go:122] "RemoveContainer" containerID="35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d"
Nov 03 07:35:39.280527 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:35:39.280509    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d\": container with ID starting with 35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d not found: ID does not exist" containerID="35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d"
Nov 03 07:35:39.280577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.280534    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d"} err="failed to get container status \"35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d\": rpc error: code = NotFound desc = could not find container \"35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d\": container with ID starting with 35460c960a3cb1c2d4a228302f5376f345d998faaa536182fafb40b20908aa9d not found: ID does not exist"
Nov 03 07:35:39.443723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.443691    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"]
Nov 03 07:35:39.449486 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.449461    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf"]
Nov 03 07:35:39.639595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.639557    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"]
Nov 03 07:35:39.640643 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.640614    4470 cpu_manager.go:414] "containerMap: removing container" podUID="b889dc0f-b0f6-4757-bc23-d983977bd559" containerName="revoke-keys"
Nov 03 07:35:39.640643 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.640641    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b889dc0f-b0f6-4757-bc23-d983977bd559" containerName="revoke-keys"
Nov 03 07:35:39.640822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.640683    4470 cpu_manager.go:414] "containerMap: removing container" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy"
Nov 03 07:35:39.640822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.640692    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy"
Nov 03 07:35:39.640902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.640831    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="b889dc0f-b0f6-4757-bc23-d983977bd559" containerName="revoke-keys"
Nov 03 07:35:39.640902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.640844    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy"
Nov 03 07:35:39.646690 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.646667    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:39.702188 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.702151    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"]
Nov 03 07:35:39.702465 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.702441    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:35:39.702719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.702695    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:35:39.702890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.702692    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:35:39.702890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.702715    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:35:39.786368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.786317    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:39.786566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.786436    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmg76\" (UniqueName: \"kubernetes.io/projected/cb415689-ab76-4649-9324-9ddf9f306bd4-kube-api-access-zmg76\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:39.786566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.786517    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:39.812150 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.812104    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" path="/var/lib/kubelet/pods/eabd31f7-12a2-4753-8a37-bc835e84615d/volumes"
Nov 03 07:35:39.888128 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.888093    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:39.888380 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.888176    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmg76\" (UniqueName: \"kubernetes.io/projected/cb415689-ab76-4649-9324-9ddf9f306bd4-kube-api-access-zmg76\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:39.888380 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.888261    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:39.888830 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.888808    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:39.890801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.890744    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:39.952040 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.952005    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmg76\" (UniqueName: \"kubernetes.io/projected/cb415689-ab76-4649-9324-9ddf9f306bd4-kube-api-access-zmg76\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:39.961856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:39.961829    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:40.348750 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:40.348047    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"]
Nov 03 07:35:41.292840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:41.292801    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l" event={"ID":"cb415689-ab76-4649-9324-9ddf9f306bd4","Type":"ContainerStarted","Data":"b19711f889fbc8253ce23ab97cc8037d1884188ba27386e2657506f4d7a1a23e"}
Nov 03 07:35:41.293242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:41.292862    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l" event={"ID":"cb415689-ab76-4649-9324-9ddf9f306bd4","Type":"ContainerStarted","Data":"e05f1cc2e30be8afe221fe6ab155c408463dee0757fba348818c8d08822dc8c5"}
Nov 03 07:35:41.425010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:41.424961    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l" podStartSLOduration=2.424948083 podStartE2EDuration="2.424948083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:35:41.424544793 +0000 UTC m=+2193.756913787" watchObservedRunningTime="2026-11-03 07:35:41.424948083 +0000 UTC m=+2193.757317076"
Nov 03 07:35:42.303954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:42.303921    4470 generic.go:356] "Generic (PLEG): container finished" podID="cb415689-ab76-4649-9324-9ddf9f306bd4" containerID="b19711f889fbc8253ce23ab97cc8037d1884188ba27386e2657506f4d7a1a23e" exitCode=0
Nov 03 07:35:42.304398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:42.304058    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l" event={"ID":"cb415689-ab76-4649-9324-9ddf9f306bd4","Type":"ContainerDied","Data":"b19711f889fbc8253ce23ab97cc8037d1884188ba27386e2657506f4d7a1a23e"}
Nov 03 07:35:42.814727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:42.814691    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"]
Nov 03 07:35:42.818351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:42.818324    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-09fd61-fr2gq"]
Nov 03 07:35:43.522872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:43.522846    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:43.693883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:43.693849    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb415689-ab76-4649-9324-9ddf9f306bd4-kube-api-access-zmg76\" (UniqueName: \"kubernetes.io/projected/cb415689-ab76-4649-9324-9ddf9f306bd4-kube-api-access-zmg76\") pod \"cb415689-ab76-4649-9324-9ddf9f306bd4\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") "
Nov 03 07:35:43.694089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:43.693934    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-xks-ca\") pod \"cb415689-ab76-4649-9324-9ddf9f306bd4\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") "
Nov 03 07:35:43.694089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:43.693961    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-openshift-ca\") pod \"cb415689-ab76-4649-9324-9ddf9f306bd4\" (UID: \"cb415689-ab76-4649-9324-9ddf9f306bd4\") "
Nov 03 07:35:43.694433 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:43.694404    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-openshift-ca" pod "cb415689-ab76-4649-9324-9ddf9f306bd4" (UID: "cb415689-ab76-4649-9324-9ddf9f306bd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:35:43.694823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:43.694771    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:43.697195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:43.697167    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb415689-ab76-4649-9324-9ddf9f306bd4-kube-api-access-zmg76" pod "cb415689-ab76-4649-9324-9ddf9f306bd4" (UID: "cb415689-ab76-4649-9324-9ddf9f306bd4"). InnerVolumeSpecName "kube-api-access-zmg76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:35:43.697323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:43.697303    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-xks-ca" pod "cb415689-ab76-4649-9324-9ddf9f306bd4" (UID: "cb415689-ab76-4649-9324-9ddf9f306bd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:35:43.796391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:43.796349    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmg76\" (UniqueName: \"kubernetes.io/projected/cb415689-ab76-4649-9324-9ddf9f306bd4-kube-api-access-zmg76\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:43.796391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:43.796386    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb415689-ab76-4649-9324-9ddf9f306bd4-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:43.807897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:43.807853    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71eb7f69-9eed-45da-a5e3-d696cbe525cb" path="/var/lib/kubelet/pods/71eb7f69-9eed-45da-a5e3-d696cbe525cb/volumes"
Nov 03 07:35:44.321898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:44.321866    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"
Nov 03 07:35:44.322076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:44.321902    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l" event={"ID":"cb415689-ab76-4649-9324-9ddf9f306bd4","Type":"ContainerDied","Data":"e05f1cc2e30be8afe221fe6ab155c408463dee0757fba348818c8d08822dc8c5"}
Nov 03 07:35:44.322076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:44.321950    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05f1cc2e30be8afe221fe6ab155c408463dee0757fba348818c8d08822dc8c5"
Nov 03 07:35:46.240922 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:46.240888    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"]
Nov 03 07:35:50.456607 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:50.454525    4470 patch_prober.go:29] interesting pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:35:50.456607 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:50.454625    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:35:51.242230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.242177    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz" podUID="3f8c9423-852c-40f5-b69d-6338dc0d310f" containerName="payload-processing" containerID="cri-o://b16cdb1d107a543281314a3f2e8566f573574a11fbf4f8d67d307d7e78cfb0f2" gracePeriod=25
Nov 03 07:35:51.252113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.252085    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"]
Nov 03 07:35:51.365118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.365082    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"]
Nov 03 07:35:51.377196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.377163    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz" event={"ID":"3f8c9423-852c-40f5-b69d-6338dc0d310f","Type":"ContainerDied","Data":"b16cdb1d107a543281314a3f2e8566f573574a11fbf4f8d67d307d7e78cfb0f2"}
Nov 03 07:35:51.377341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.377169    4470 generic.go:356] "Generic (PLEG): container finished" podID="3f8c9423-852c-40f5-b69d-6338dc0d310f" containerID="b16cdb1d107a543281314a3f2e8566f573574a11fbf4f8d67d307d7e78cfb0f2" exitCode=0
Nov 03 07:35:51.378979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.378958    4470 generic.go:356] "Generic (PLEG): container finished" podID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerID="622390a52f6b3bbec04dea77086726ac2bbe536be61cdcf9922027aceca5088d" exitCode=0
Nov 03 07:35:51.379149 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.379042    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" event={"ID":"05c084dd-3680-4f60-80a1-a1b35e7a15ce","Type":"ContainerDied","Data":"622390a52f6b3bbec04dea77086726ac2bbe536be61cdcf9922027aceca5088d"}
Nov 03 07:35:51.379633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.379615    4470 scope.go:122] "RemoveContainer" containerID="622390a52f6b3bbec04dea77086726ac2bbe536be61cdcf9922027aceca5088d"
Nov 03 07:35:51.381496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.381468    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:35:51.381560 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.381533    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:35:51.381601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.381567    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:35:51.538894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.538844    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"]
Nov 03 07:35:51.541261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.540365    4470 cpu_manager.go:414] "containerMap: removing container" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy"
Nov 03 07:35:51.541261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.540390    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy"
Nov 03 07:35:51.541261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.540446    4470 cpu_manager.go:414] "containerMap: removing container" podUID="cb415689-ab76-4649-9324-9ddf9f306bd4" containerName="revoke-keys"
Nov 03 07:35:51.541261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.540455    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb415689-ab76-4649-9324-9ddf9f306bd4" containerName="revoke-keys"
Nov 03 07:35:51.541261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.540682    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabd31f7-12a2-4753-8a37-bc835e84615d" containerName="istio-proxy"
Nov 03 07:35:51.541261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.540706    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb415689-ab76-4649-9324-9ddf9f306bd4" containerName="revoke-keys"
Nov 03 07:35:51.550589 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.548240    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:51.590115 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.590085    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"]
Nov 03 07:35:51.599688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.599650    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:35:51.604118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.604088    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:35:51.604324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.604305    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:35:51.604385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.604372    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:35:51.604474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.604458    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:35:51.654196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.652610    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8c9423-852c-40f5-b69d-6338dc0d310f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8c9423-852c-40f5-b69d-6338dc0d310f-plugins-config-volume\") pod \"3f8c9423-852c-40f5-b69d-6338dc0d310f\" (UID: \"3f8c9423-852c-40f5-b69d-6338dc0d310f\") "
Nov 03 07:35:51.654196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.652804    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8c9423-852c-40f5-b69d-6338dc0d310f-kube-api-access-r4km5\" (UniqueName: \"kubernetes.io/projected/3f8c9423-852c-40f5-b69d-6338dc0d310f-kube-api-access-r4km5\") pod \"3f8c9423-852c-40f5-b69d-6338dc0d310f\" (UID: \"3f8c9423-852c-40f5-b69d-6338dc0d310f\") "
Nov 03 07:35:51.654196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.653507    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8c9423-852c-40f5-b69d-6338dc0d310f-plugins-config-volume" pod "3f8c9423-852c-40f5-b69d-6338dc0d310f" (UID: "3f8c9423-852c-40f5-b69d-6338dc0d310f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:35:51.654196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.653850    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:51.654196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.654071    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:51.654619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.654232    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc9ks\" (UniqueName: \"kubernetes.io/projected/78154f35-159b-4166-840f-25ba3dedc95a-kube-api-access-pc9ks\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:51.654619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.654355    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8c9423-852c-40f5-b69d-6338dc0d310f-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:51.657239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.657211    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8c9423-852c-40f5-b69d-6338dc0d310f-kube-api-access-r4km5" pod "3f8c9423-852c-40f5-b69d-6338dc0d310f" (UID: "3f8c9423-852c-40f5-b69d-6338dc0d310f"). InnerVolumeSpecName "kube-api-access-r4km5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:35:51.756102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.756065    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:51.756292 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.756240    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:51.756363 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.756331    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc9ks\" (UniqueName: \"kubernetes.io/projected/78154f35-159b-4166-840f-25ba3dedc95a-kube-api-access-pc9ks\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:51.756444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.756428    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4km5\" (UniqueName: \"kubernetes.io/projected/3f8c9423-852c-40f5-b69d-6338dc0d310f-kube-api-access-r4km5\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:51.756772 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.756749    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:51.758785 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.758761    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:51.864212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.863695    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc9ks\" (UniqueName: \"kubernetes.io/projected/78154f35-159b-4166-840f-25ba3dedc95a-kube-api-access-pc9ks\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:51.910565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:51.910527    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:52.303437 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:52.303402    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"]
Nov 03 07:35:52.389164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:52.389120    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"
Nov 03 07:35:52.396316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:52.389203    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz" event={"ID":"3f8c9423-852c-40f5-b69d-6338dc0d310f","Type":"ContainerDied","Data":"60086db41c6371996bd390bfd09bfb550b87f3ad683619b9f59a68d6527cdf01"}
Nov 03 07:35:52.396316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:52.389270    4470 scope.go:122] "RemoveContainer" containerID="b16cdb1d107a543281314a3f2e8566f573574a11fbf4f8d67d307d7e78cfb0f2"
Nov 03 07:35:52.396316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:52.392271    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" event={"ID":"05c084dd-3680-4f60-80a1-a1b35e7a15ce","Type":"ContainerStarted","Data":"cc919a1204d3019437ecac8025c379bf8e8c248cb8478cefca8e1a1d10c1f160"}
Nov 03 07:35:52.396316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:52.394665    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2" event={"ID":"78154f35-159b-4166-840f-25ba3dedc95a","Type":"ContainerStarted","Data":"bf2de21bd866e29c369cbec5783cff310599f5bf1b07e2d5daf59d96a884e0cd"}
Nov 03 07:35:52.698381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:52.698339    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"]
Nov 03 07:35:52.703920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:52.703887    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz"]
Nov 03 07:35:53.169159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:53.168565    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:35:53.169159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:53.168680    4470 patch_prober.go:29] interesting pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 07:35:53.169159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:53.168720    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 07:35:53.403593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:53.403563    4470 generic.go:356] "Generic (PLEG): container finished" podID="78154f35-159b-4166-840f-25ba3dedc95a" containerID="26f9a6a4e68b1734a9ac7568214acd030e7e002363f7710227547efcd5b50e2d" exitCode=0
Nov 03 07:35:53.403763 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:53.403654    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2" event={"ID":"78154f35-159b-4166-840f-25ba3dedc95a","Type":"ContainerDied","Data":"26f9a6a4e68b1734a9ac7568214acd030e7e002363f7710227547efcd5b50e2d"}
Nov 03 07:35:53.859157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:53.857556    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8c9423-852c-40f5-b69d-6338dc0d310f" path="/var/lib/kubelet/pods/3f8c9423-852c-40f5-b69d-6338dc0d310f/volumes"
Nov 03 07:35:54.172334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.172248    4470 patch_prober.go:29] interesting pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 07:35:54.172334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.172317    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 07:35:54.611813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.611789    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:54.728480 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.728444    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78154f35-159b-4166-840f-25ba3dedc95a-kube-api-access-pc9ks\" (UniqueName: \"kubernetes.io/projected/78154f35-159b-4166-840f-25ba3dedc95a-kube-api-access-pc9ks\") pod \"78154f35-159b-4166-840f-25ba3dedc95a\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") "
Nov 03 07:35:54.728767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.728750    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-openshift-ca\") pod \"78154f35-159b-4166-840f-25ba3dedc95a\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") "
Nov 03 07:35:54.728843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.728795    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-xks-ca\") pod \"78154f35-159b-4166-840f-25ba3dedc95a\" (UID: \"78154f35-159b-4166-840f-25ba3dedc95a\") "
Nov 03 07:35:54.729127 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.729105    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-openshift-ca" pod "78154f35-159b-4166-840f-25ba3dedc95a" (UID: "78154f35-159b-4166-840f-25ba3dedc95a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:35:54.729430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.729411    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:54.730826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.730789    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78154f35-159b-4166-840f-25ba3dedc95a-kube-api-access-pc9ks" pod "78154f35-159b-4166-840f-25ba3dedc95a" (UID: "78154f35-159b-4166-840f-25ba3dedc95a"). InnerVolumeSpecName "kube-api-access-pc9ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:35:54.731094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.731075    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-xks-ca" pod "78154f35-159b-4166-840f-25ba3dedc95a" (UID: "78154f35-159b-4166-840f-25ba3dedc95a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:35:54.830771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.830742    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc9ks\" (UniqueName: \"kubernetes.io/projected/78154f35-159b-4166-840f-25ba3dedc95a-kube-api-access-pc9ks\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:54.830771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:54.830771    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78154f35-159b-4166-840f-25ba3dedc95a-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:55.164670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:55.164575    4470 patch_prober.go:29] interesting pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 07:35:55.165280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:55.165242    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 07:35:55.421114 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:55.421029    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"
Nov 03 07:35:55.421114 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:55.421069    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2" event={"ID":"78154f35-159b-4166-840f-25ba3dedc95a","Type":"ContainerDied","Data":"bf2de21bd866e29c369cbec5783cff310599f5bf1b07e2d5daf59d96a884e0cd"}
Nov 03 07:35:55.421371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:55.421173    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf2de21bd866e29c369cbec5783cff310599f5bf1b07e2d5daf59d96a884e0cd"
Nov 03 07:35:55.511152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:55.511113    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"]
Nov 03 07:35:55.538487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:55.538460    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72a642-5nsm2"]
Nov 03 07:35:55.809465 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:55.809386    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31049885-e9d4-41a0-8b46-c5caa7a0e2c8" path="/var/lib/kubelet/pods/31049885-e9d4-41a0-8b46-c5caa7a0e2c8/volumes"
Nov 03 07:35:56.177161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.170539    4470 patch_prober.go:29] interesting pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 07:35:56.177161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.170599    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 07:35:56.267295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.267249    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk" podUID="a8b9a208-a748-4e5f-a83e-b075f08bd3a2" containerName="payload-processing" containerID="cri-o://6af27f1014f3c2f66022674c116102f4390592dbeb1d23b03e4d131642551911" gracePeriod=25
Nov 03 07:35:56.371172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.368204    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b" podUID="091fa31f-5c4c-4817-bc3d-0c36edcac9c3" containerName="payload-pre-processing" containerID="cri-o://795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45" gracePeriod=25
Nov 03 07:35:56.437207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.432445    4470 generic.go:356] "Generic (PLEG): container finished" podID="a8b9a208-a748-4e5f-a83e-b075f08bd3a2" containerID="6af27f1014f3c2f66022674c116102f4390592dbeb1d23b03e4d131642551911" exitCode=0
Nov 03 07:35:56.437207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.432505    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk" event={"ID":"a8b9a208-a748-4e5f-a83e-b075f08bd3a2","Type":"ContainerDied","Data":"6af27f1014f3c2f66022674c116102f4390592dbeb1d23b03e4d131642551911"}
Nov 03 07:35:56.617826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.617799    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:35:56.778864 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.778820    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:35:56.796445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.796409    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-plugins-config-volume\") pod \"a8b9a208-a748-4e5f-a83e-b075f08bd3a2\" (UID: \"a8b9a208-a748-4e5f-a83e-b075f08bd3a2\") "
Nov 03 07:35:56.796595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.796483    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-kube-api-access-rcr65\" (UniqueName: \"kubernetes.io/projected/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-kube-api-access-rcr65\") pod \"a8b9a208-a748-4e5f-a83e-b075f08bd3a2\" (UID: \"a8b9a208-a748-4e5f-a83e-b075f08bd3a2\") "
Nov 03 07:35:56.796758 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.796738    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-plugins-config-volume" pod "a8b9a208-a748-4e5f-a83e-b075f08bd3a2" (UID: "a8b9a208-a748-4e5f-a83e-b075f08bd3a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:35:56.796989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.796976    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:56.798702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.798682    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-kube-api-access-rcr65" pod "a8b9a208-a748-4e5f-a83e-b075f08bd3a2" (UID: "a8b9a208-a748-4e5f-a83e-b075f08bd3a2"). InnerVolumeSpecName "kube-api-access-rcr65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:35:56.897861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.897825    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-kube-api-access-r99xn\" (UniqueName: \"kubernetes.io/projected/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-kube-api-access-r99xn\") pod \"091fa31f-5c4c-4817-bc3d-0c36edcac9c3\" (UID: \"091fa31f-5c4c-4817-bc3d-0c36edcac9c3\") "
Nov 03 07:35:56.898063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.897923    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-plugins-config-volume\") pod \"091fa31f-5c4c-4817-bc3d-0c36edcac9c3\" (UID: \"091fa31f-5c4c-4817-bc3d-0c36edcac9c3\") "
Nov 03 07:35:56.899341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.899313    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcr65\" (UniqueName: \"kubernetes.io/projected/a8b9a208-a748-4e5f-a83e-b075f08bd3a2-kube-api-access-rcr65\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:56.899715 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.899689    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-plugins-config-volume" pod "091fa31f-5c4c-4817-bc3d-0c36edcac9c3" (UID: "091fa31f-5c4c-4817-bc3d-0c36edcac9c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:35:56.901534 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:56.901509    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-kube-api-access-r99xn" pod "091fa31f-5c4c-4817-bc3d-0c36edcac9c3" (UID: "091fa31f-5c4c-4817-bc3d-0c36edcac9c3"). InnerVolumeSpecName "kube-api-access-r99xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:35:57.000447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.000366    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r99xn\" (UniqueName: \"kubernetes.io/projected/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-kube-api-access-r99xn\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:57.000447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.000395    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091fa31f-5c4c-4817-bc3d-0c36edcac9c3-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:35:57.168160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.162473    4470 patch_prober.go:29] interesting pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 07:35:57.168160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.162533    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 07:35:57.441346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.441315    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"
Nov 03 07:35:57.441819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.441342    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk" event={"ID":"a8b9a208-a748-4e5f-a83e-b075f08bd3a2","Type":"ContainerDied","Data":"d9f9b583f1dfc79ee166fb3074407de925f1d8ad22a3f4331676780cf128e205"}
Nov 03 07:35:57.441819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.441404    4470 scope.go:122] "RemoveContainer" containerID="6af27f1014f3c2f66022674c116102f4390592dbeb1d23b03e4d131642551911"
Nov 03 07:35:57.442853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.442782    4470 generic.go:356] "Generic (PLEG): container finished" podID="091fa31f-5c4c-4817-bc3d-0c36edcac9c3" containerID="795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45" exitCode=0
Nov 03 07:35:57.442853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.442817    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b" event={"ID":"091fa31f-5c4c-4817-bc3d-0c36edcac9c3","Type":"ContainerDied","Data":"795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45"}
Nov 03 07:35:57.442853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.442832    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"
Nov 03 07:35:57.443047 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.442875    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b" event={"ID":"091fa31f-5c4c-4817-bc3d-0c36edcac9c3","Type":"ContainerDied","Data":"3f4fe8f9fa457792f06c02f9532826f8e19b51267d0326f2eda535537b5c9ae4"}
Nov 03 07:35:57.478677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.478646    4470 scope.go:122] "RemoveContainer" containerID="795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45"
Nov 03 07:35:57.490929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.490904    4470 scope.go:122] "RemoveContainer" containerID="795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45"
Nov 03 07:35:57.491251 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:35:57.491229    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45\": container with ID starting with 795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45 not found: ID does not exist" containerID="795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45"
Nov 03 07:35:57.491315 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.491261    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45"} err="failed to get container status \"795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45\": rpc error: code = NotFound desc = could not find container \"795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45\": container with ID starting with 795574b656b9210e3b3fd90ffc9e839d0b8e180643decf5facec669b45f30d45 not found: ID does not exist"
Nov 03 07:35:57.630860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.630824    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"]
Nov 03 07:35:57.642754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.642714    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk"]
Nov 03 07:35:57.771082 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.771035    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"]
Nov 03 07:35:57.774612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.774577    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b"]
Nov 03 07:35:57.807576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.807535    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091fa31f-5c4c-4817-bc3d-0c36edcac9c3" path="/var/lib/kubelet/pods/091fa31f-5c4c-4817-bc3d-0c36edcac9c3/volumes"
Nov 03 07:35:57.807970 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:57.807955    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b9a208-a748-4e5f-a83e-b075f08bd3a2" path="/var/lib/kubelet/pods/a8b9a208-a748-4e5f-a83e-b075f08bd3a2/volumes"
Nov 03 07:35:58.166461 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:58.166434    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:35:58.166834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:58.166815    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:35:58.167537 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:35:58.167517    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:36:03.673598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:03.673563    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"]
Nov 03 07:36:03.674036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:03.673774    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy" containerID="cri-o://55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c" gracePeriod=30
Nov 03 07:36:04.762291 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:36:04.762254    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:36:05.788240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.788202    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"]
Nov 03 07:36:05.788836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.788820    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3f8c9423-852c-40f5-b69d-6338dc0d310f" containerName="payload-processing"
Nov 03 07:36:05.788898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.788836    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8c9423-852c-40f5-b69d-6338dc0d310f" containerName="payload-processing"
Nov 03 07:36:05.788898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.788861    4470 cpu_manager.go:414] "containerMap: removing container" podUID="78154f35-159b-4166-840f-25ba3dedc95a" containerName="revoke-keys"
Nov 03 07:36:05.788898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.788866    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78154f35-159b-4166-840f-25ba3dedc95a" containerName="revoke-keys"
Nov 03 07:36:05.788898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.788887    4470 cpu_manager.go:414] "containerMap: removing container" podUID="091fa31f-5c4c-4817-bc3d-0c36edcac9c3" containerName="payload-pre-processing"
Nov 03 07:36:05.788898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.788892    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091fa31f-5c4c-4817-bc3d-0c36edcac9c3" containerName="payload-pre-processing"
Nov 03 07:36:05.789066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.788912    4470 cpu_manager.go:414] "containerMap: removing container" podUID="a8b9a208-a748-4e5f-a83e-b075f08bd3a2" containerName="payload-processing"
Nov 03 07:36:05.789066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.788917    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b9a208-a748-4e5f-a83e-b075f08bd3a2" containerName="payload-processing"
Nov 03 07:36:05.789066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.789007    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="091fa31f-5c4c-4817-bc3d-0c36edcac9c3" containerName="payload-pre-processing"
Nov 03 07:36:05.789066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.789018    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b9a208-a748-4e5f-a83e-b075f08bd3a2" containerName="payload-processing"
Nov 03 07:36:05.789066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.789025    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8c9423-852c-40f5-b69d-6338dc0d310f" containerName="payload-processing"
Nov 03 07:36:05.789066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.789034    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="78154f35-159b-4166-840f-25ba3dedc95a" containerName="revoke-keys"
Nov 03 07:36:05.792625 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.792610    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:05.850686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.850656    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"]
Nov 03 07:36:05.860577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.860545    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246s8\" (UniqueName: \"kubernetes.io/projected/5243dca6-cb46-4df4-bb32-f90a09845e6a-kube-api-access-246s8\") pod \"maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:05.860894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.860872    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:05.861037 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.860939    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:05.896498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.896472    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:36:05.896706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.896520    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:36:05.896706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.896563    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:36:05.897224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.897205    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:36:05.961946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.961913    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:05.962175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.961972    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:05.962175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.962050    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246s8\" (UniqueName: \"kubernetes.io/projected/5243dca6-cb46-4df4-bb32-f90a09845e6a-kube-api-access-246s8\") pod \"maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:05.962812 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.962727    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:05.964512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:05.964489    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:06.026268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.026237    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-246s8\" (UniqueName: \"kubernetes.io/projected/5243dca6-cb46-4df4-bb32-f90a09845e6a-kube-api-access-246s8\") pod \"maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:06.107209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.107174    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:06.167850 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.166629    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"]
Nov 03 07:36:06.172541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.172514    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:06.229773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.229705    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"]
Nov 03 07:36:06.266905 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.266869    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:06.267082 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.267043    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk46p\" (UniqueName: \"kubernetes.io/projected/e9c92b07-a24e-48c4-9654-ef2d7432ba35-kube-api-access-jk46p\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:06.267123 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.267082    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:06.368869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.368786    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk46p\" (UniqueName: \"kubernetes.io/projected/e9c92b07-a24e-48c4-9654-ef2d7432ba35-kube-api-access-jk46p\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:06.368869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.368837    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:06.369124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.368951    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:06.369581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.369556    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:06.371528 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.371503    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:06.433038 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.433007    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk46p\" (UniqueName: \"kubernetes.io/projected/e9c92b07-a24e-48c4-9654-ef2d7432ba35-kube-api-access-jk46p\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:06.493023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.493000    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:06.496014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.495989    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"]
Nov 03 07:36:06.496245 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:36:06.496210    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5243dca6_cb46_4df4_bb32_f90a09845e6a.slice/crio-c442c46c436d0e155d4af9b7b8d011edd22da26c4c7578161903c8729f33dc51 WatchSource:0}: Error finding container c442c46c436d0e155d4af9b7b8d011edd22da26c4c7578161903c8729f33dc51: Status 404 returned error can't find the container with id c442c46c436d0e155d4af9b7b8d011edd22da26c4c7578161903c8729f33dc51
Nov 03 07:36:06.576773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.576743    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf" event={"ID":"5243dca6-cb46-4df4-bb32-f90a09845e6a","Type":"ContainerStarted","Data":"c442c46c436d0e155d4af9b7b8d011edd22da26c4c7578161903c8729f33dc51"}
Nov 03 07:36:06.875408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:06.875381    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"]
Nov 03 07:36:06.875742 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:36:06.875595    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c92b07_a24e_48c4_9654_ef2d7432ba35.slice/crio-e89c3b41c48b626c326048c63dbe89c08831b6741b9fdb06dbed9bfefe228908 WatchSource:0}: Error finding container e89c3b41c48b626c326048c63dbe89c08831b6741b9fdb06dbed9bfefe228908: Status 404 returned error can't find the container with id e89c3b41c48b626c326048c63dbe89c08831b6741b9fdb06dbed9bfefe228908
Nov 03 07:36:07.586214 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:07.586171    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs" event={"ID":"e9c92b07-a24e-48c4-9654-ef2d7432ba35","Type":"ContainerStarted","Data":"a040bf53e0f88c1ed1a552026bbff9f9e3f2b62c6e918756c1bc35e36a369f28"}
Nov 03 07:36:07.586406 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:07.586294    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs" event={"ID":"e9c92b07-a24e-48c4-9654-ef2d7432ba35","Type":"ContainerStarted","Data":"e89c3b41c48b626c326048c63dbe89c08831b6741b9fdb06dbed9bfefe228908"}
Nov 03 07:36:07.587752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:07.587726    4470 generic.go:356] "Generic (PLEG): container finished" podID="5243dca6-cb46-4df4-bb32-f90a09845e6a" containerID="8edfe2ade876f5d47e87438b43148d1b2e90f96e0a16876c63528c1272c803e0" exitCode=0
Nov 03 07:36:07.587867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:07.587758    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf" event={"ID":"5243dca6-cb46-4df4-bb32-f90a09845e6a","Type":"ContainerDied","Data":"8edfe2ade876f5d47e87438b43148d1b2e90f96e0a16876c63528c1272c803e0"}
Nov 03 07:36:07.707243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:07.707144    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs" podStartSLOduration=1.707116252 podStartE2EDuration="1.707116252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:36:07.705430007 +0000 UTC m=+2220.037799002" watchObservedRunningTime="2026-11-03 07:36:07.707116252 +0000 UTC m=+2220.039485246"
Nov 03 07:36:08.607476 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:08.607440    4470 generic.go:356] "Generic (PLEG): container finished" podID="e9c92b07-a24e-48c4-9654-ef2d7432ba35" containerID="a040bf53e0f88c1ed1a552026bbff9f9e3f2b62c6e918756c1bc35e36a369f28" exitCode=0
Nov 03 07:36:08.607909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:08.607527    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs" event={"ID":"e9c92b07-a24e-48c4-9654-ef2d7432ba35","Type":"ContainerDied","Data":"a040bf53e0f88c1ed1a552026bbff9f9e3f2b62c6e918756c1bc35e36a369f28"}
Nov 03 07:36:08.772928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:08.772905    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:08.910050 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:08.909969    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-xks-ca\") pod \"5243dca6-cb46-4df4-bb32-f90a09845e6a\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") "
Nov 03 07:36:08.910256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:08.910083    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5243dca6-cb46-4df4-bb32-f90a09845e6a-kube-api-access-246s8\" (UniqueName: \"kubernetes.io/projected/5243dca6-cb46-4df4-bb32-f90a09845e6a-kube-api-access-246s8\") pod \"5243dca6-cb46-4df4-bb32-f90a09845e6a\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") "
Nov 03 07:36:08.910256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:08.910178    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-openshift-ca\") pod \"5243dca6-cb46-4df4-bb32-f90a09845e6a\" (UID: \"5243dca6-cb46-4df4-bb32-f90a09845e6a\") "
Nov 03 07:36:08.910675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:08.910651    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-openshift-ca" pod "5243dca6-cb46-4df4-bb32-f90a09845e6a" (UID: "5243dca6-cb46-4df4-bb32-f90a09845e6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:36:08.911016 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:08.910992    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:08.912389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:08.912367    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-xks-ca" pod "5243dca6-cb46-4df4-bb32-f90a09845e6a" (UID: "5243dca6-cb46-4df4-bb32-f90a09845e6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:36:08.912892 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:08.912875    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5243dca6-cb46-4df4-bb32-f90a09845e6a-kube-api-access-246s8" pod "5243dca6-cb46-4df4-bb32-f90a09845e6a" (UID: "5243dca6-cb46-4df4-bb32-f90a09845e6a"). InnerVolumeSpecName "kube-api-access-246s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:09.012419 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.012386    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5243dca6-cb46-4df4-bb32-f90a09845e6a-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.012419 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.012420    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-246s8\" (UniqueName: \"kubernetes.io/projected/5243dca6-cb46-4df4-bb32-f90a09845e6a-kube-api-access-246s8\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.375678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.375645    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:36:09.519157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519052    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-socket\") pod \"c4854012-52e4-4b86-81e9-3af28f40a8a1\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") "
Nov 03 07:36:09.519338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519125    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-credential-socket\") pod \"c4854012-52e4-4b86-81e9-3af28f40a8a1\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") "
Nov 03 07:36:09.519338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519259    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-envoy\") pod \"c4854012-52e4-4b86-81e9-3af28f40a8a1\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") "
Nov 03 07:36:09.519338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519316    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-data\") pod \"c4854012-52e4-4b86-81e9-3af28f40a8a1\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") "
Nov 03 07:36:09.519513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519384    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-socket" pod "c4854012-52e4-4b86-81e9-3af28f40a8a1" (UID: "c4854012-52e4-4b86-81e9-3af28f40a8a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:09.519513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519399    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-credential-socket" pod "c4854012-52e4-4b86-81e9-3af28f40a8a1" (UID: "c4854012-52e4-4b86-81e9-3af28f40a8a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:09.519513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519447    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-certs\") pod \"c4854012-52e4-4b86-81e9-3af28f40a8a1\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") "
Nov 03 07:36:09.519513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519489    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-kube-api-access-vlb4c\" (UniqueName: \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-kube-api-access-vlb4c\") pod \"c4854012-52e4-4b86-81e9-3af28f40a8a1\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") "
Nov 03 07:36:09.519745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519574    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4854012-52e4-4b86-81e9-3af28f40a8a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4854012-52e4-4b86-81e9-3af28f40a8a1-istiod-ca-cert\") pod \"c4854012-52e4-4b86-81e9-3af28f40a8a1\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") "
Nov 03 07:36:09.519745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519584    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-data" pod "c4854012-52e4-4b86-81e9-3af28f40a8a1" (UID: "c4854012-52e4-4b86-81e9-3af28f40a8a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:09.519745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519634    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-token\" (UniqueName: \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-token\") pod \"c4854012-52e4-4b86-81e9-3af28f40a8a1\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") "
Nov 03 07:36:09.519745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519690    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-certs" pod "c4854012-52e4-4b86-81e9-3af28f40a8a1" (UID: "c4854012-52e4-4b86-81e9-3af28f40a8a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:09.519964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519750    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-podinfo\") pod \"c4854012-52e4-4b86-81e9-3af28f40a8a1\" (UID: \"c4854012-52e4-4b86-81e9-3af28f40a8a1\") "
Nov 03 07:36:09.519964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.519929    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4854012-52e4-4b86-81e9-3af28f40a8a1-istiod-ca-cert" pod "c4854012-52e4-4b86-81e9-3af28f40a8a1" (UID: "c4854012-52e4-4b86-81e9-3af28f40a8a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:36:09.520647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.520617    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4854012-52e4-4b86-81e9-3af28f40a8a1-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.520797 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.520650    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.520797 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.520667    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.520797 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.520681    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.520797 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.520695    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.521655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.521628    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-envoy" pod "c4854012-52e4-4b86-81e9-3af28f40a8a1" (UID: "c4854012-52e4-4b86-81e9-3af28f40a8a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:09.521920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.521892    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-token" pod "c4854012-52e4-4b86-81e9-3af28f40a8a1" (UID: "c4854012-52e4-4b86-81e9-3af28f40a8a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:09.522014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.521919    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-kube-api-access-vlb4c" pod "c4854012-52e4-4b86-81e9-3af28f40a8a1" (UID: "c4854012-52e4-4b86-81e9-3af28f40a8a1"). InnerVolumeSpecName "kube-api-access-vlb4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:09.522014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.521970    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-podinfo" pod "c4854012-52e4-4b86-81e9-3af28f40a8a1" (UID: "c4854012-52e4-4b86-81e9-3af28f40a8a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:36:09.618154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.618105    4470 generic.go:356] "Generic (PLEG): container finished" podID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerID="55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c" exitCode=0
Nov 03 07:36:09.618626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.618169    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"
Nov 03 07:36:09.618626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.618165    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" event={"ID":"c4854012-52e4-4b86-81e9-3af28f40a8a1","Type":"ContainerDied","Data":"55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c"}
Nov 03 07:36:09.618626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.618229    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2" event={"ID":"c4854012-52e4-4b86-81e9-3af28f40a8a1","Type":"ContainerDied","Data":"344b01f1503830cccc3abd3427cbe098aef7557c2269d4c98e48c7bea48a6d44"}
Nov 03 07:36:09.618626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.618256    4470 scope.go:122] "RemoveContainer" containerID="55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c"
Nov 03 07:36:09.619897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.619878    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"
Nov 03 07:36:09.620036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.619883    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf" event={"ID":"5243dca6-cb46-4df4-bb32-f90a09845e6a","Type":"ContainerDied","Data":"c442c46c436d0e155d4af9b7b8d011edd22da26c4c7578161903c8729f33dc51"}
Nov 03 07:36:09.620036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.619936    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c442c46c436d0e155d4af9b7b8d011edd22da26c4c7578161903c8729f33dc51"
Nov 03 07:36:09.621696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.621673    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.621814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.621701    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.621814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.621717    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4854012-52e4-4b86-81e9-3af28f40a8a1-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.621814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.621730    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlb4c\" (UniqueName: \"kubernetes.io/projected/c4854012-52e4-4b86-81e9-3af28f40a8a1-kube-api-access-vlb4c\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.634823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.634696    4470 scope.go:122] "RemoveContainer" containerID="3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388"
Nov 03 07:36:09.646558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.646537    4470 scope.go:122] "RemoveContainer" containerID="55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c"
Nov 03 07:36:09.646900 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:36:09.646879    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c\": container with ID starting with 55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c not found: ID does not exist" containerID="55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c"
Nov 03 07:36:09.646982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.646907    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c"} err="failed to get container status \"55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c\": rpc error: code = NotFound desc = could not find container \"55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c\": container with ID starting with 55f7f201e6fc452e9378f48bf09a84b3b98fea83d471b6b320b081cafe1eaf2c not found: ID does not exist"
Nov 03 07:36:09.646982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.646927    4470 scope.go:122] "RemoveContainer" containerID="3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388"
Nov 03 07:36:09.647218 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:36:09.647198    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388\": container with ID starting with 3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388 not found: ID does not exist" containerID="3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388"
Nov 03 07:36:09.647311 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.647223    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388"} err="failed to get container status \"3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388\": rpc error: code = NotFound desc = could not find container \"3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388\": container with ID starting with 3b78c4ba7e0695217043753935dd345b92c92631b370c85362963977168a3388 not found: ID does not exist"
Nov 03 07:36:09.789534 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.789471    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:09.792063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.792042    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"]
Nov 03 07:36:09.797975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.797950    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2"]
Nov 03 07:36:09.807642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.807617    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" path="/var/lib/kubelet/pods/c4854012-52e4-4b86-81e9-3af28f40a8a1/volumes"
Nov 03 07:36:09.925069 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.924990    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-openshift-ca\") pod \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") "
Nov 03 07:36:09.925313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.925197    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c92b07-a24e-48c4-9654-ef2d7432ba35-kube-api-access-jk46p\" (UniqueName: \"kubernetes.io/projected/e9c92b07-a24e-48c4-9654-ef2d7432ba35-kube-api-access-jk46p\") pod \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") "
Nov 03 07:36:09.925388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.925316    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-xks-ca\") pod \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\" (UID: \"e9c92b07-a24e-48c4-9654-ef2d7432ba35\") "
Nov 03 07:36:09.925479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.925457    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-openshift-ca" pod "e9c92b07-a24e-48c4-9654-ef2d7432ba35" (UID: "e9c92b07-a24e-48c4-9654-ef2d7432ba35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:36:09.925922 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.925903    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:09.927411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.927387    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c92b07-a24e-48c4-9654-ef2d7432ba35-kube-api-access-jk46p" pod "e9c92b07-a24e-48c4-9654-ef2d7432ba35" (UID: "e9c92b07-a24e-48c4-9654-ef2d7432ba35"). InnerVolumeSpecName "kube-api-access-jk46p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:09.927510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:09.927487    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-xks-ca" pod "e9c92b07-a24e-48c4-9654-ef2d7432ba35" (UID: "e9c92b07-a24e-48c4-9654-ef2d7432ba35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:36:10.027310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:10.027277    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c92b07-a24e-48c4-9654-ef2d7432ba35-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:10.027310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:10.027305    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk46p\" (UniqueName: \"kubernetes.io/projected/e9c92b07-a24e-48c4-9654-ef2d7432ba35-kube-api-access-jk46p\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:10.630339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:10.630310    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"
Nov 03 07:36:10.630731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:10.630346    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs" event={"ID":"e9c92b07-a24e-48c4-9654-ef2d7432ba35","Type":"ContainerDied","Data":"e89c3b41c48b626c326048c63dbe89c08831b6741b9fdb06dbed9bfefe228908"}
Nov 03 07:36:10.630731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:10.630389    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e89c3b41c48b626c326048c63dbe89c08831b6741b9fdb06dbed9bfefe228908"
Nov 03 07:36:14.015779 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:14.015740    4470 scope.go:122] "RemoveContainer" containerID="b54b4569e09e50310a941394c08d862cb4856bda44b65c4b51c76a5b1674f164"
Nov 03 07:36:18.242578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:18.242542    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"]
Nov 03 07:36:18.355305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:18.355269    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"]
Nov 03 07:36:20.753793 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:20.753761    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"]
Nov 03 07:36:20.867028 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:20.866991    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"]
Nov 03 07:36:23.057071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.057019    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"]
Nov 03 07:36:23.107514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.107488    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a271ff-glzf2"]
Nov 03 07:36:23.242930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.242884    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv" podUID="3581dd25-7393-4efb-989e-1c44e94263e5" containerName="payload-processing" containerID="cri-o://c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6" gracePeriod=25
Nov 03 07:36:23.355616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.355577    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl" podUID="81e93120-dd39-49a0-8b5c-1b394ac78cbb" containerName="payload-pre-processing" containerID="cri-o://bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937" gracePeriod=25
Nov 03 07:36:23.537792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.537754    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:36:23.641557 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.641530    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:36:23.648867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.648847    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3581dd25-7393-4efb-989e-1c44e94263e5-kube-api-access-k4hkl\" (UniqueName: \"kubernetes.io/projected/3581dd25-7393-4efb-989e-1c44e94263e5-kube-api-access-k4hkl\") pod \"3581dd25-7393-4efb-989e-1c44e94263e5\" (UID: \"3581dd25-7393-4efb-989e-1c44e94263e5\") "
Nov 03 07:36:23.648981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.648945    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3581dd25-7393-4efb-989e-1c44e94263e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3581dd25-7393-4efb-989e-1c44e94263e5-plugins-config-volume\") pod \"3581dd25-7393-4efb-989e-1c44e94263e5\" (UID: \"3581dd25-7393-4efb-989e-1c44e94263e5\") "
Nov 03 07:36:23.649324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.649294    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3581dd25-7393-4efb-989e-1c44e94263e5-plugins-config-volume" pod "3581dd25-7393-4efb-989e-1c44e94263e5" (UID: "3581dd25-7393-4efb-989e-1c44e94263e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:36:23.649444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.649423    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3581dd25-7393-4efb-989e-1c44e94263e5-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:23.650975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.650951    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3581dd25-7393-4efb-989e-1c44e94263e5-kube-api-access-k4hkl" pod "3581dd25-7393-4efb-989e-1c44e94263e5" (UID: "3581dd25-7393-4efb-989e-1c44e94263e5"). InnerVolumeSpecName "kube-api-access-k4hkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:23.750268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.750240    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e93120-dd39-49a0-8b5c-1b394ac78cbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e93120-dd39-49a0-8b5c-1b394ac78cbb-plugins-config-volume\") pod \"81e93120-dd39-49a0-8b5c-1b394ac78cbb\" (UID: \"81e93120-dd39-49a0-8b5c-1b394ac78cbb\") "
Nov 03 07:36:23.750445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.750429    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e93120-dd39-49a0-8b5c-1b394ac78cbb-kube-api-access-lzxb7\" (UniqueName: \"kubernetes.io/projected/81e93120-dd39-49a0-8b5c-1b394ac78cbb-kube-api-access-lzxb7\") pod \"81e93120-dd39-49a0-8b5c-1b394ac78cbb\" (UID: \"81e93120-dd39-49a0-8b5c-1b394ac78cbb\") "
Nov 03 07:36:23.750617 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.750595    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e93120-dd39-49a0-8b5c-1b394ac78cbb-plugins-config-volume" pod "81e93120-dd39-49a0-8b5c-1b394ac78cbb" (UID: "81e93120-dd39-49a0-8b5c-1b394ac78cbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:36:23.750876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.750853    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e93120-dd39-49a0-8b5c-1b394ac78cbb-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:23.750876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.750871    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4hkl\" (UniqueName: \"kubernetes.io/projected/3581dd25-7393-4efb-989e-1c44e94263e5-kube-api-access-k4hkl\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:23.752592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.752572    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e93120-dd39-49a0-8b5c-1b394ac78cbb-kube-api-access-lzxb7" pod "81e93120-dd39-49a0-8b5c-1b394ac78cbb" (UID: "81e93120-dd39-49a0-8b5c-1b394ac78cbb"). InnerVolumeSpecName "kube-api-access-lzxb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:23.762960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.762932    4470 generic.go:356] "Generic (PLEG): container finished" podID="81e93120-dd39-49a0-8b5c-1b394ac78cbb" containerID="bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937" exitCode=0
Nov 03 07:36:23.763090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.762978    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"
Nov 03 07:36:23.763090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.763016    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl" event={"ID":"81e93120-dd39-49a0-8b5c-1b394ac78cbb","Type":"ContainerDied","Data":"bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937"}
Nov 03 07:36:23.763090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.763069    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl" event={"ID":"81e93120-dd39-49a0-8b5c-1b394ac78cbb","Type":"ContainerDied","Data":"bda859be07331038f9365702d76dac9ce6ce1a14cb7fc97e343974824410eed9"}
Nov 03 07:36:23.763234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.763092    4470 scope.go:122] "RemoveContainer" containerID="bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937"
Nov 03 07:36:23.764600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.764581    4470 generic.go:356] "Generic (PLEG): container finished" podID="3581dd25-7393-4efb-989e-1c44e94263e5" containerID="c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6" exitCode=0
Nov 03 07:36:23.764703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.764610    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv" event={"ID":"3581dd25-7393-4efb-989e-1c44e94263e5","Type":"ContainerDied","Data":"c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6"}
Nov 03 07:36:23.764703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.764626    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"
Nov 03 07:36:23.764703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.764658    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv" event={"ID":"3581dd25-7393-4efb-989e-1c44e94263e5","Type":"ContainerDied","Data":"9d8d243deee730c49c0c6e008a0b52c6a2ea862b2f22cf41cc2e6dcbae6a6b52"}
Nov 03 07:36:23.780758 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.780736    4470 scope.go:122] "RemoveContainer" containerID="bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937"
Nov 03 07:36:23.781184 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:36:23.781120    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937\": container with ID starting with bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937 not found: ID does not exist" containerID="bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937"
Nov 03 07:36:23.781284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.781197    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937"} err="failed to get container status \"bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937\": rpc error: code = NotFound desc = could not find container \"bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937\": container with ID starting with bf28fc3311851e986784adc22e0f0126d5e64b00f55dd01ef0230042bde0a937 not found: ID does not exist"
Nov 03 07:36:23.781284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.781228    4470 scope.go:122] "RemoveContainer" containerID="c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6"
Nov 03 07:36:23.793928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.793905    4470 scope.go:122] "RemoveContainer" containerID="c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6"
Nov 03 07:36:23.794338 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:36:23.794313    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6\": container with ID starting with c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6 not found: ID does not exist" containerID="c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6"
Nov 03 07:36:23.794427 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.794344    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6"} err="failed to get container status \"c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6\": rpc error: code = NotFound desc = could not find container \"c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6\": container with ID starting with c850977f5e44aa8ff875e28d9821e25e46444a50e60a4508608703472f0892f6 not found: ID does not exist"
Nov 03 07:36:23.809548 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.809521    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e4621a-80bc-4277-a35c-503e84e16875" path="/var/lib/kubelet/pods/34e4621a-80bc-4277-a35c-503e84e16875/volumes"
Nov 03 07:36:23.852123 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.852091    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzxb7\" (UniqueName: \"kubernetes.io/projected/81e93120-dd39-49a0-8b5c-1b394ac78cbb-kube-api-access-lzxb7\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:23.897428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.897345    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"]
Nov 03 07:36:23.905098 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:23.905070    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl"]
Nov 03 07:36:24.016906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:24.016876    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"]
Nov 03 07:36:24.020388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:24.020360    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv"]
Nov 03 07:36:25.754521 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:25.754462    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k" podUID="11c19799-37af-4990-b6df-c1d93cafaf93" containerName="payload-processing" containerID="cri-o://5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d" gracePeriod=25
Nov 03 07:36:25.807599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:25.807566    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3581dd25-7393-4efb-989e-1c44e94263e5" path="/var/lib/kubelet/pods/3581dd25-7393-4efb-989e-1c44e94263e5/volumes"
Nov 03 07:36:25.808060 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:25.808045    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e93120-dd39-49a0-8b5c-1b394ac78cbb" path="/var/lib/kubelet/pods/81e93120-dd39-49a0-8b5c-1b394ac78cbb/volumes"
Nov 03 07:36:25.867522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:25.867476    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt" podUID="ecc50dc9-fdea-459b-9cc1-db1aa822aac9" containerName="payload-pre-processing" containerID="cri-o://e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984" gracePeriod=25
Nov 03 07:36:26.096201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.096044    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:36:26.184181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.184127    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c19799-37af-4990-b6df-c1d93cafaf93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c19799-37af-4990-b6df-c1d93cafaf93-plugins-config-volume\") pod \"11c19799-37af-4990-b6df-c1d93cafaf93\" (UID: \"11c19799-37af-4990-b6df-c1d93cafaf93\") "
Nov 03 07:36:26.184323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.184195    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c19799-37af-4990-b6df-c1d93cafaf93-kube-api-access-5rb7k\" (UniqueName: \"kubernetes.io/projected/11c19799-37af-4990-b6df-c1d93cafaf93-kube-api-access-5rb7k\") pod \"11c19799-37af-4990-b6df-c1d93cafaf93\" (UID: \"11c19799-37af-4990-b6df-c1d93cafaf93\") "
Nov 03 07:36:26.184566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.184541    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c19799-37af-4990-b6df-c1d93cafaf93-plugins-config-volume" pod "11c19799-37af-4990-b6df-c1d93cafaf93" (UID: "11c19799-37af-4990-b6df-c1d93cafaf93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:36:26.184769 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.184749    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c19799-37af-4990-b6df-c1d93cafaf93-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:26.186527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.186504    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c19799-37af-4990-b6df-c1d93cafaf93-kube-api-access-5rb7k" pod "11c19799-37af-4990-b6df-c1d93cafaf93" (UID: "11c19799-37af-4990-b6df-c1d93cafaf93"). InnerVolumeSpecName "kube-api-access-5rb7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:26.187651 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.187634    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:36:26.285147 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.285109    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-plugins-config-volume\") pod \"ecc50dc9-fdea-459b-9cc1-db1aa822aac9\" (UID: \"ecc50dc9-fdea-459b-9cc1-db1aa822aac9\") "
Nov 03 07:36:26.285347 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.285291    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-kube-api-access-cm9ds\" (UniqueName: \"kubernetes.io/projected/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-kube-api-access-cm9ds\") pod \"ecc50dc9-fdea-459b-9cc1-db1aa822aac9\" (UID: \"ecc50dc9-fdea-459b-9cc1-db1aa822aac9\") "
Nov 03 07:36:26.285559 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.285530    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-plugins-config-volume" pod "ecc50dc9-fdea-459b-9cc1-db1aa822aac9" (UID: "ecc50dc9-fdea-459b-9cc1-db1aa822aac9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:36:26.285791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.285774    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:26.285852 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.285797    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rb7k\" (UniqueName: \"kubernetes.io/projected/11c19799-37af-4990-b6df-c1d93cafaf93-kube-api-access-5rb7k\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:26.287467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.287445    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-kube-api-access-cm9ds" pod "ecc50dc9-fdea-459b-9cc1-db1aa822aac9" (UID: "ecc50dc9-fdea-459b-9cc1-db1aa822aac9"). InnerVolumeSpecName "kube-api-access-cm9ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:26.386949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.386873    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm9ds\" (UniqueName: \"kubernetes.io/projected/ecc50dc9-fdea-459b-9cc1-db1aa822aac9-kube-api-access-cm9ds\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:26.790017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.789987    4470 generic.go:356] "Generic (PLEG): container finished" podID="11c19799-37af-4990-b6df-c1d93cafaf93" containerID="5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d" exitCode=0
Nov 03 07:36:26.790513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.790039    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"
Nov 03 07:36:26.790513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.790079    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k" event={"ID":"11c19799-37af-4990-b6df-c1d93cafaf93","Type":"ContainerDied","Data":"5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d"}
Nov 03 07:36:26.790513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.790170    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k" event={"ID":"11c19799-37af-4990-b6df-c1d93cafaf93","Type":"ContainerDied","Data":"978b6c92ae18031506ef7a0fa562a797ea4422f8da39b40c02cec34ece6a96f3"}
Nov 03 07:36:26.790513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.790208    4470 scope.go:122] "RemoveContainer" containerID="5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d"
Nov 03 07:36:26.791823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.791707    4470 generic.go:356] "Generic (PLEG): container finished" podID="ecc50dc9-fdea-459b-9cc1-db1aa822aac9" containerID="e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984" exitCode=0
Nov 03 07:36:26.791823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.791758    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"
Nov 03 07:36:26.791823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.791800    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt" event={"ID":"ecc50dc9-fdea-459b-9cc1-db1aa822aac9","Type":"ContainerDied","Data":"e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984"}
Nov 03 07:36:26.792010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.791855    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt" event={"ID":"ecc50dc9-fdea-459b-9cc1-db1aa822aac9","Type":"ContainerDied","Data":"754f303132d3b1fdad0019f5c3be971dfe157be6f3049b9bc4ec972db97fed7a"}
Nov 03 07:36:26.801707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.801685    4470 scope.go:122] "RemoveContainer" containerID="5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d"
Nov 03 07:36:26.802006 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:36:26.801978    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d\": container with ID starting with 5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d not found: ID does not exist" containerID="5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d"
Nov 03 07:36:26.802099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.802016    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d"} err="failed to get container status \"5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d\": rpc error: code = NotFound desc = could not find container \"5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d\": container with ID starting with 5c46bcfd87583467665748e73cf7c417e417d6d5acb0bd323585ac4df869348d not found: ID does not exist"
Nov 03 07:36:26.802099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.802037    4470 scope.go:122] "RemoveContainer" containerID="e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984"
Nov 03 07:36:26.812827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.812810    4470 scope.go:122] "RemoveContainer" containerID="e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984"
Nov 03 07:36:26.813116 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:36:26.813097    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984\": container with ID starting with e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984 not found: ID does not exist" containerID="e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984"
Nov 03 07:36:26.813214 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.813125    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984"} err="failed to get container status \"e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984\": rpc error: code = NotFound desc = could not find container \"e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984\": container with ID starting with e04d6a8f6fbe3ea5f674d7b5cf1c5c0c92bfeb861f8a7571a7f79ab299a45984 not found: ID does not exist"
Nov 03 07:36:26.962494 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:26.962466    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"]
Nov 03 07:36:27.013449 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:27.013421    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt"]
Nov 03 07:36:27.081744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:27.081713    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"]
Nov 03 07:36:27.089930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:27.089905    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k"]
Nov 03 07:36:27.816325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:27.816289    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c19799-37af-4990-b6df-c1d93cafaf93" path="/var/lib/kubelet/pods/11c19799-37af-4990-b6df-c1d93cafaf93/volumes"
Nov 03 07:36:27.816859 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:27.816839    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc50dc9-fdea-459b-9cc1-db1aa822aac9" path="/var/lib/kubelet/pods/ecc50dc9-fdea-459b-9cc1-db1aa822aac9/volumes"
Nov 03 07:36:34.760899 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:36:34.760865    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:36:45.614628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:45.614595    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"]
Nov 03 07:36:45.615291 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:45.614877    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy" containerID="cri-o://dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77" gracePeriod=30
Nov 03 07:36:46.179388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:46.179351    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"]
Nov 03 07:36:46.179595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:46.179562    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerName="istio-proxy" containerID="cri-o://a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2" gracePeriod=30
Nov 03 07:36:46.912503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:46.912472    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:36:46.912974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:46.912539    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:36:47.697557 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:47.697520    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"]
Nov 03 07:36:47.745160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:47.745091    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:36:47.745327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:47.745182    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:36:47.746811 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:47.746790    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-893d83-4dsgs"]
Nov 03 07:36:47.809286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:47.809250    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c92b07-a24e-48c4-9654-ef2d7432ba35" path="/var/lib/kubelet/pods/e9c92b07-a24e-48c4-9654-ef2d7432ba35/volumes"
Nov 03 07:36:50.887921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.887851    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:36:50.963367 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963337    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-token\" (UniqueName: \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-token\") pod \"8a39c230-3238-4a0d-89c4-073b1ba5a429\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") "
Nov 03 07:36:50.963542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963383    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-podinfo\") pod \"8a39c230-3238-4a0d-89c4-073b1ba5a429\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") "
Nov 03 07:36:50.963542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963409    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-socket\") pod \"8a39c230-3238-4a0d-89c4-073b1ba5a429\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") "
Nov 03 07:36:50.963542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963437    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-data\") pod \"8a39c230-3238-4a0d-89c4-073b1ba5a429\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") "
Nov 03 07:36:50.963542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963491    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a39c230-3238-4a0d-89c4-073b1ba5a429-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a39c230-3238-4a0d-89c4-073b1ba5a429-istiod-ca-cert\") pod \"8a39c230-3238-4a0d-89c4-073b1ba5a429\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") "
Nov 03 07:36:50.963785 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963574    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-credential-socket\") pod \"8a39c230-3238-4a0d-89c4-073b1ba5a429\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") "
Nov 03 07:36:50.963785 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963646    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-certs\") pod \"8a39c230-3238-4a0d-89c4-073b1ba5a429\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") "
Nov 03 07:36:50.963785 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963738    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-envoy\") pod \"8a39c230-3238-4a0d-89c4-073b1ba5a429\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") "
Nov 03 07:36:50.963785 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963751    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-socket" pod "8a39c230-3238-4a0d-89c4-073b1ba5a429" (UID: "8a39c230-3238-4a0d-89c4-073b1ba5a429"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:50.963995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963799    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-kube-api-access-v89rs\" (UniqueName: \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-kube-api-access-v89rs\") pod \"8a39c230-3238-4a0d-89c4-073b1ba5a429\" (UID: \"8a39c230-3238-4a0d-89c4-073b1ba5a429\") "
Nov 03 07:36:50.963995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963778    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-data" pod "8a39c230-3238-4a0d-89c4-073b1ba5a429" (UID: "8a39c230-3238-4a0d-89c4-073b1ba5a429"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:50.963995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963927    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a39c230-3238-4a0d-89c4-073b1ba5a429-istiod-ca-cert" pod "8a39c230-3238-4a0d-89c4-073b1ba5a429" (UID: "8a39c230-3238-4a0d-89c4-073b1ba5a429"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:36:50.963995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.963929    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-credential-socket" pod "8a39c230-3238-4a0d-89c4-073b1ba5a429" (UID: "8a39c230-3238-4a0d-89c4-073b1ba5a429"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:50.964309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.964174    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-certs" pod "8a39c230-3238-4a0d-89c4-073b1ba5a429" (UID: "8a39c230-3238-4a0d-89c4-073b1ba5a429"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:50.964464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.964441    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:50.964796 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.964771    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:50.964914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.964798    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a39c230-3238-4a0d-89c4-073b1ba5a429-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:50.964914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.964816    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:50.964914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.964833    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:50.965837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.965809    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-token" pod "8a39c230-3238-4a0d-89c4-073b1ba5a429" (UID: "8a39c230-3238-4a0d-89c4-073b1ba5a429"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:50.965924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.965876    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-podinfo" pod "8a39c230-3238-4a0d-89c4-073b1ba5a429" (UID: "8a39c230-3238-4a0d-89c4-073b1ba5a429"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:36:50.966183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.966159    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-kube-api-access-v89rs" pod "8a39c230-3238-4a0d-89c4-073b1ba5a429" (UID: "8a39c230-3238-4a0d-89c4-073b1ba5a429"). InnerVolumeSpecName "kube-api-access-v89rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:50.966351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:50.966335    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-envoy" pod "8a39c230-3238-4a0d-89c4-073b1ba5a429" (UID: "8a39c230-3238-4a0d-89c4-073b1ba5a429"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:51.019174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.019147    4470 generic.go:356] "Generic (PLEG): container finished" podID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerID="dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77" exitCode=0
Nov 03 07:36:51.019325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.019188    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"
Nov 03 07:36:51.019325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.019238    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" event={"ID":"8a39c230-3238-4a0d-89c4-073b1ba5a429","Type":"ContainerDied","Data":"dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77"}
Nov 03 07:36:51.019325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.019292    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59" event={"ID":"8a39c230-3238-4a0d-89c4-073b1ba5a429","Type":"ContainerDied","Data":"192817ed497e0e43810e12cef01fbd111be6e02efedaa9291b6800a37824a66b"}
Nov 03 07:36:51.019325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.019319    4470 scope.go:122] "RemoveContainer" containerID="dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77"
Nov 03 07:36:51.030202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.030182    4470 scope.go:122] "RemoveContainer" containerID="dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77"
Nov 03 07:36:51.030499 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:36:51.030477    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77\": container with ID starting with dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77 not found: ID does not exist" containerID="dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77"
Nov 03 07:36:51.030567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.030509    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77"} err="failed to get container status \"dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77\": rpc error: code = NotFound desc = could not find container \"dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77\": container with ID starting with dd58e8825340f312855797a49f203bb7c96e15c082cda45797fe7451bb309f77 not found: ID does not exist"
Nov 03 07:36:51.066858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.066818    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.066858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.066849    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89rs\" (UniqueName: \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-kube-api-access-v89rs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.066858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.066860    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.067112 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.066869    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a39c230-3238-4a0d-89c4-073b1ba5a429-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.183106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.183021    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"]
Nov 03 07:36:51.191450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.191423    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59"]
Nov 03 07:36:51.434791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.434730    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:36:51.574122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574081    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-credential-socket\") pod \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") "
Nov 03 07:36:51.574330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574189    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-socket\") pod \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") "
Nov 03 07:36:51.574330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574233    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-envoy\") pod \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") "
Nov 03 07:36:51.574330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574304    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-token\" (UniqueName: \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-token\") pod \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") "
Nov 03 07:36:51.574514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574347    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-kube-api-access-vzt8j\" (UniqueName: \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-kube-api-access-vzt8j\") pod \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") "
Nov 03 07:36:51.574514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574413    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-credential-socket" pod "79cad3f3-671b-47dd-8cb9-98d9e4b2439b" (UID: "79cad3f3-671b-47dd-8cb9-98d9e4b2439b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:51.574514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574455    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-podinfo\") pod \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") "
Nov 03 07:36:51.574680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574544    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-socket" pod "79cad3f3-671b-47dd-8cb9-98d9e4b2439b" (UID: "79cad3f3-671b-47dd-8cb9-98d9e4b2439b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:51.574680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574570    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-certs\") pod \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") "
Nov 03 07:36:51.574680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574649    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istiod-ca-cert\") pod \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") "
Nov 03 07:36:51.574837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574755    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-data\") pod \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\" (UID: \"79cad3f3-671b-47dd-8cb9-98d9e4b2439b\") "
Nov 03 07:36:51.574942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.574920    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-certs" pod "79cad3f3-671b-47dd-8cb9-98d9e4b2439b" (UID: "79cad3f3-671b-47dd-8cb9-98d9e4b2439b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:51.575085 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.575058    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-data" pod "79cad3f3-671b-47dd-8cb9-98d9e4b2439b" (UID: "79cad3f3-671b-47dd-8cb9-98d9e4b2439b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:51.575194 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.575104    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istiod-ca-cert" pod "79cad3f3-671b-47dd-8cb9-98d9e4b2439b" (UID: "79cad3f3-671b-47dd-8cb9-98d9e4b2439b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:36:51.575562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.575538    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.575674 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.575566    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.575674 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.575581    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.575674 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.575594    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.575674 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.575608    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.577340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.577312    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-podinfo" pod "79cad3f3-671b-47dd-8cb9-98d9e4b2439b" (UID: "79cad3f3-671b-47dd-8cb9-98d9e4b2439b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:36:51.577529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.577508    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-kube-api-access-vzt8j" pod "79cad3f3-671b-47dd-8cb9-98d9e4b2439b" (UID: "79cad3f3-671b-47dd-8cb9-98d9e4b2439b"). InnerVolumeSpecName "kube-api-access-vzt8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:51.577828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.577808    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-envoy" pod "79cad3f3-671b-47dd-8cb9-98d9e4b2439b" (UID: "79cad3f3-671b-47dd-8cb9-98d9e4b2439b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:36:51.578204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.578179    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-token" pod "79cad3f3-671b-47dd-8cb9-98d9e4b2439b" (UID: "79cad3f3-671b-47dd-8cb9-98d9e4b2439b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:36:51.676925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.676888    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.676925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.676926    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.677176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.676941    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzt8j\" (UniqueName: \"kubernetes.io/projected/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-kube-api-access-vzt8j\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.677176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.676953    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cad3f3-671b-47dd-8cb9-98d9e4b2439b-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:36:51.807886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:51.807806    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" path="/var/lib/kubelet/pods/8a39c230-3238-4a0d-89c4-073b1ba5a429/volumes"
Nov 03 07:36:52.037723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:52.037686    4470 generic.go:356] "Generic (PLEG): container finished" podID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerID="a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2" exitCode=0
Nov 03 07:36:52.037723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:52.037726    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" event={"ID":"79cad3f3-671b-47dd-8cb9-98d9e4b2439b","Type":"ContainerDied","Data":"a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2"}
Nov 03 07:36:52.038279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:52.037738    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"
Nov 03 07:36:52.038279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:52.037799    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr" event={"ID":"79cad3f3-671b-47dd-8cb9-98d9e4b2439b","Type":"ContainerDied","Data":"e6db7cc96f365fd86c94d061dda773e34c55890c00ab05b4c7fe9c27ada0f8b5"}
Nov 03 07:36:52.038279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:52.037823    4470 scope.go:122] "RemoveContainer" containerID="a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2"
Nov 03 07:36:52.047984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:52.047962    4470 scope.go:122] "RemoveContainer" containerID="a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2"
Nov 03 07:36:52.048269 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:36:52.048249    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2\": container with ID starting with a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2 not found: ID does not exist" containerID="a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2"
Nov 03 07:36:52.048338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:52.048277    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2"} err="failed to get container status \"a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2\": rpc error: code = NotFound desc = could not find container \"a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2\": container with ID starting with a0686b104f3e36c4f8cd7acea99b452a500d20991d39fa435c6725334f866fe2 not found: ID does not exist"
Nov 03 07:36:52.215368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:52.215333    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"]
Nov 03 07:36:52.220951 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:52.220921    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr"]
Nov 03 07:36:53.807432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:53.807403    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" path="/var/lib/kubelet/pods/79cad3f3-671b-47dd-8cb9-98d9e4b2439b/volumes"
Nov 03 07:36:53.961664 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:53.961629    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"]
Nov 03 07:36:54.012079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:54.012046    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-32b500-8wmj4"]
Nov 03 07:36:55.808195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:55.808163    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b889dc0f-b0f6-4757-bc23-d983977bd559" path="/var/lib/kubelet/pods/b889dc0f-b0f6-4757-bc23-d983977bd559/volumes"
Nov 03 07:36:57.270666 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.270627    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"]
Nov 03 07:36:57.271613 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271593    4470 cpu_manager.go:414] "containerMap: removing container" podUID="81e93120-dd39-49a0-8b5c-1b394ac78cbb" containerName="payload-pre-processing"
Nov 03 07:36:57.271677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271617    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e93120-dd39-49a0-8b5c-1b394ac78cbb" containerName="payload-pre-processing"
Nov 03 07:36:57.271677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271633    4470 cpu_manager.go:414] "containerMap: removing container" podUID="11c19799-37af-4990-b6df-c1d93cafaf93" containerName="payload-processing"
Nov 03 07:36:57.271677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271641    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c19799-37af-4990-b6df-c1d93cafaf93" containerName="payload-processing"
Nov 03 07:36:57.271790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271692    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ecc50dc9-fdea-459b-9cc1-db1aa822aac9" containerName="payload-pre-processing"
Nov 03 07:36:57.271790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271701    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc50dc9-fdea-459b-9cc1-db1aa822aac9" containerName="payload-pre-processing"
Nov 03 07:36:57.271790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271721    4470 cpu_manager.go:414] "containerMap: removing container" podUID="5243dca6-cb46-4df4-bb32-f90a09845e6a" containerName="revoke-keys"
Nov 03 07:36:57.271790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271729    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5243dca6-cb46-4df4-bb32-f90a09845e6a" containerName="revoke-keys"
Nov 03 07:36:57.271790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271746    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy"
Nov 03 07:36:57.271790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271753    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy"
Nov 03 07:36:57.271790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271775    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy"
Nov 03 07:36:57.271790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271782    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271796    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3581dd25-7393-4efb-989e-1c44e94263e5" containerName="payload-processing"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271802    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3581dd25-7393-4efb-989e-1c44e94263e5" containerName="payload-processing"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271820    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e9c92b07-a24e-48c4-9654-ef2d7432ba35" containerName="revoke-keys"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271827    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c92b07-a24e-48c4-9654-ef2d7432ba35" containerName="revoke-keys"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271837    4470 cpu_manager.go:414] "containerMap: removing container" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerName="istio-proxy"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271844    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerName="istio-proxy"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271986    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.271999    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a39c230-3238-4a0d-89c4-073b1ba5a429" containerName="istio-proxy"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.272011    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3581dd25-7393-4efb-989e-1c44e94263e5" containerName="payload-processing"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.272024    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c92b07-a24e-48c4-9654-ef2d7432ba35" containerName="revoke-keys"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.272032    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="79cad3f3-671b-47dd-8cb9-98d9e4b2439b" containerName="istio-proxy"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.272048    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e93120-dd39-49a0-8b5c-1b394ac78cbb" containerName="payload-pre-processing"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.272057    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c19799-37af-4990-b6df-c1d93cafaf93" containerName="payload-processing"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.272069    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc50dc9-fdea-459b-9cc1-db1aa822aac9" containerName="payload-pre-processing"
Nov 03 07:36:57.272180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.272078    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="5243dca6-cb46-4df4-bb32-f90a09845e6a" containerName="revoke-keys"
Nov 03 07:36:57.277166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.277148    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:36:57.333435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.333403    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"]
Nov 03 07:36:57.377802 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.377767    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:36:57.377980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.377841    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6kvf\" (UniqueName: \"kubernetes.io/projected/545e8729-76b0-473c-8bbf-2e651e972d52-kube-api-access-r6kvf\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:36:57.378045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.378020    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:36:57.379664 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.379625    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:36:57.379788 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.379679    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:36:57.379788 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.379708    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:36:57.382488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.382470    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:36:57.479057 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.479028    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:36:57.479259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.479089    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6kvf\" (UniqueName: \"kubernetes.io/projected/545e8729-76b0-473c-8bbf-2e651e972d52-kube-api-access-r6kvf\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:36:57.479259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.479189    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:36:57.479841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.479818    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:36:57.481599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.481570    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:36:57.543120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.543045    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6kvf\" (UniqueName: \"kubernetes.io/projected/545e8729-76b0-473c-8bbf-2e651e972d52-kube-api-access-r6kvf\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:36:57.591674 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.591632    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:36:57.991841 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:36:57.991810    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545e8729_76b0_473c_8bbf_2e651e972d52.slice/crio-b05a83383a47704d608d7607e04c22082bbc7092c950cd01321e4007dc81cea1 WatchSource:0}: Error finding container b05a83383a47704d608d7607e04c22082bbc7092c950cd01321e4007dc81cea1: Status 404 returned error can't find the container with id b05a83383a47704d608d7607e04c22082bbc7092c950cd01321e4007dc81cea1
Nov 03 07:36:57.992632 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:57.992604    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"]
Nov 03 07:36:58.091471 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:58.091436    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4" event={"ID":"545e8729-76b0-473c-8bbf-2e651e972d52","Type":"ContainerStarted","Data":"b05a83383a47704d608d7607e04c22082bbc7092c950cd01321e4007dc81cea1"}
Nov 03 07:36:59.102337 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:59.102305    4470 generic.go:356] "Generic (PLEG): container finished" podID="545e8729-76b0-473c-8bbf-2e651e972d52" containerID="901373d68f69089078a9d5964bca2f973991c6564ac38dde3c0a2161aef9fbd2" exitCode=0
Nov 03 07:36:59.102766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:36:59.102372    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4" event={"ID":"545e8729-76b0-473c-8bbf-2e651e972d52","Type":"ContainerDied","Data":"901373d68f69089078a9d5964bca2f973991c6564ac38dde3c0a2161aef9fbd2"}
Nov 03 07:37:00.275681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:00.275658    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:37:00.455552 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:00.455460    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545e8729-76b0-473c-8bbf-2e651e972d52-kube-api-access-r6kvf\" (UniqueName: \"kubernetes.io/projected/545e8729-76b0-473c-8bbf-2e651e972d52-kube-api-access-r6kvf\") pod \"545e8729-76b0-473c-8bbf-2e651e972d52\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") "
Nov 03 07:37:00.455755 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:00.455736    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-openshift-ca\") pod \"545e8729-76b0-473c-8bbf-2e651e972d52\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") "
Nov 03 07:37:00.455828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:00.455790    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-xks-ca\") pod \"545e8729-76b0-473c-8bbf-2e651e972d52\" (UID: \"545e8729-76b0-473c-8bbf-2e651e972d52\") "
Nov 03 07:37:00.456201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:00.456115    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-openshift-ca" pod "545e8729-76b0-473c-8bbf-2e651e972d52" (UID: "545e8729-76b0-473c-8bbf-2e651e972d52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:37:00.456462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:00.456414    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:00.457901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:00.457872    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545e8729-76b0-473c-8bbf-2e651e972d52-kube-api-access-r6kvf" pod "545e8729-76b0-473c-8bbf-2e651e972d52" (UID: "545e8729-76b0-473c-8bbf-2e651e972d52"). InnerVolumeSpecName "kube-api-access-r6kvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:37:00.458351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:00.458331    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-xks-ca" pod "545e8729-76b0-473c-8bbf-2e651e972d52" (UID: "545e8729-76b0-473c-8bbf-2e651e972d52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:37:00.557923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:00.557882    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545e8729-76b0-473c-8bbf-2e651e972d52-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:00.558079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:00.557941    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6kvf\" (UniqueName: \"kubernetes.io/projected/545e8729-76b0-473c-8bbf-2e651e972d52-kube-api-access-r6kvf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:00.999270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:00.999237    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"]
Nov 03 07:37:01.020838 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:01.020801    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1fa30f-fjr4l"]
Nov 03 07:37:01.120816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:01.120782    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4" event={"ID":"545e8729-76b0-473c-8bbf-2e651e972d52","Type":"ContainerDied","Data":"b05a83383a47704d608d7607e04c22082bbc7092c950cd01321e4007dc81cea1"}
Nov 03 07:37:01.121009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:01.120833    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"
Nov 03 07:37:01.121009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:01.120834    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b05a83383a47704d608d7607e04c22082bbc7092c950cd01321e4007dc81cea1"
Nov 03 07:37:01.810951 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:01.810870    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb415689-ab76-4649-9324-9ddf9f306bd4" path="/var/lib/kubelet/pods/cb415689-ab76-4649-9324-9ddf9f306bd4/volumes"
Nov 03 07:37:04.781126 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:37:04.781093    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:37:07.984887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:07.984794    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"]
Nov 03 07:37:07.984887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:07.984848    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e6dc7-nskx2"]
Nov 03 07:37:09.318886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:09.318849    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"]
Nov 03 07:37:09.374305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:09.374271    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6d9a2-vhqbf"]
Nov 03 07:37:09.807456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:09.807376    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5243dca6-cb46-4df4-bb32-f90a09845e6a" path="/var/lib/kubelet/pods/5243dca6-cb46-4df4-bb32-f90a09845e6a/volumes"
Nov 03 07:37:09.807974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:09.807952    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78154f35-159b-4166-840f-25ba3dedc95a" path="/var/lib/kubelet/pods/78154f35-159b-4166-840f-25ba3dedc95a/volumes"
Nov 03 07:37:18.055224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:18.055185    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"]
Nov 03 07:37:18.055827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:18.055449    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy" containerID="cri-o://cc919a1204d3019437ecac8025c379bf8e8c248cb8478cefca8e1a1d10c1f160" gracePeriod=30
Nov 03 07:37:23.361983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.361493    4470 generic.go:356] "Generic (PLEG): container finished" podID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerID="cc919a1204d3019437ecac8025c379bf8e8c248cb8478cefca8e1a1d10c1f160" exitCode=0
Nov 03 07:37:23.361983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.361757    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" event={"ID":"05c084dd-3680-4f60-80a1-a1b35e7a15ce","Type":"ContainerDied","Data":"cc919a1204d3019437ecac8025c379bf8e8c248cb8478cefca8e1a1d10c1f160"}
Nov 03 07:37:23.361983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.361825    4470 scope.go:122] "RemoveContainer" containerID="622390a52f6b3bbec04dea77086726ac2bbe536be61cdcf9922027aceca5088d"
Nov 03 07:37:23.395497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.395478    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:37:23.501397 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501369    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-certs\") pod \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") "
Nov 03 07:37:23.501579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501455    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istiod-ca-cert\") pod \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") "
Nov 03 07:37:23.501579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501531    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-credential-socket\") pod \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") "
Nov 03 07:37:23.501579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501550    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-kube-api-access-6hl6c\" (UniqueName: \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-kube-api-access-6hl6c\") pod \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") "
Nov 03 07:37:23.501579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501575    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-envoy\") pod \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") "
Nov 03 07:37:23.501814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501594    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-data\") pod \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") "
Nov 03 07:37:23.501814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501655    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-podinfo\") pod \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") "
Nov 03 07:37:23.501814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501667    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-certs" pod "05c084dd-3680-4f60-80a1-a1b35e7a15ce" (UID: "05c084dd-3680-4f60-80a1-a1b35e7a15ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:37:23.501814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501749    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-token\" (UniqueName: \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-token\") pod \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") "
Nov 03 07:37:23.502036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501847    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-socket\") pod \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\" (UID: \"05c084dd-3680-4f60-80a1-a1b35e7a15ce\") "
Nov 03 07:37:23.502036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501900    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-credential-socket" pod "05c084dd-3680-4f60-80a1-a1b35e7a15ce" (UID: "05c084dd-3680-4f60-80a1-a1b35e7a15ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:37:23.502036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501898    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istiod-ca-cert" pod "05c084dd-3680-4f60-80a1-a1b35e7a15ce" (UID: "05c084dd-3680-4f60-80a1-a1b35e7a15ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:37:23.502036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.501919    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-data" pod "05c084dd-3680-4f60-80a1-a1b35e7a15ce" (UID: "05c084dd-3680-4f60-80a1-a1b35e7a15ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:37:23.502278 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.502252    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-socket" pod "05c084dd-3680-4f60-80a1-a1b35e7a15ce" (UID: "05c084dd-3680-4f60-80a1-a1b35e7a15ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:37:23.502691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.502665    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:23.502805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.502694    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:23.502805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.502709    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:23.502805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.502723    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:23.502805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.502739    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:23.504003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.503981    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-envoy" pod "05c084dd-3680-4f60-80a1-a1b35e7a15ce" (UID: "05c084dd-3680-4f60-80a1-a1b35e7a15ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:37:23.504125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.504097    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-token" pod "05c084dd-3680-4f60-80a1-a1b35e7a15ce" (UID: "05c084dd-3680-4f60-80a1-a1b35e7a15ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:37:23.504235 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.504176    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-kube-api-access-6hl6c" pod "05c084dd-3680-4f60-80a1-a1b35e7a15ce" (UID: "05c084dd-3680-4f60-80a1-a1b35e7a15ce"). InnerVolumeSpecName "kube-api-access-6hl6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:37:23.504284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.504241    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-podinfo" pod "05c084dd-3680-4f60-80a1-a1b35e7a15ce" (UID: "05c084dd-3680-4f60-80a1-a1b35e7a15ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:37:23.603633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.603599    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hl6c\" (UniqueName: \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-kube-api-access-6hl6c\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:23.603633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.603629    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:23.603633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.603640    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:23.603971 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:23.603658    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c084dd-3680-4f60-80a1-a1b35e7a15ce-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:24.402173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:24.402105    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k" event={"ID":"05c084dd-3680-4f60-80a1-a1b35e7a15ce","Type":"ContainerDied","Data":"b881ad60df653b176126da40a2b6adf3173c779a32ee68f62c47da436fccc577"}
Nov 03 07:37:24.403036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:24.402200    4470 scope.go:122] "RemoveContainer" containerID="cc919a1204d3019437ecac8025c379bf8e8c248cb8478cefca8e1a1d10c1f160"
Nov 03 07:37:24.403036 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:24.402250    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"
Nov 03 07:37:24.690885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:24.690804    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"]
Nov 03 07:37:24.696880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:24.696850    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k"]
Nov 03 07:37:25.807604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:25.807572    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" path="/var/lib/kubelet/pods/05c084dd-3680-4f60-80a1-a1b35e7a15ce/volumes"
Nov 03 07:37:34.778537 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:37:34.778502    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:37:39.784177 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.784017    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"]
Nov 03 07:37:39.786216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.786093    4470 cpu_manager.go:414] "containerMap: removing container" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy"
Nov 03 07:37:39.786393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.786376    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy"
Nov 03 07:37:39.786490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.786480    4470 cpu_manager.go:414] "containerMap: removing container" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy"
Nov 03 07:37:39.786567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.786558    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy"
Nov 03 07:37:39.786661 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.786652    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy"
Nov 03 07:37:39.786735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.786726    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy"
Nov 03 07:37:39.786830 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.786821    4470 cpu_manager.go:414] "containerMap: removing container" podUID="545e8729-76b0-473c-8bbf-2e651e972d52" containerName="revoke-keys"
Nov 03 07:37:39.786905 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.786894    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545e8729-76b0-473c-8bbf-2e651e972d52" containerName="revoke-keys"
Nov 03 07:37:39.787159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.787125    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="545e8729-76b0-473c-8bbf-2e651e972d52" containerName="revoke-keys"
Nov 03 07:37:39.787288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.787278    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy"
Nov 03 07:37:39.787381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.787371    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c084dd-3680-4f60-80a1-a1b35e7a15ce" containerName="istio-proxy"
Nov 03 07:37:39.787452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.787443    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4854012-52e4-4b86-81e9-3af28f40a8a1" containerName="istio-proxy"
Nov 03 07:37:39.796406 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.796004    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:37:39.853101 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.853058    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"]
Nov 03 07:37:39.883245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.883168    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dcdb23\""
Nov 03 07:37:39.883433 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.883307    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dcdb23-dockercfg-gsm25\""
Nov 03 07:37:39.891111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.891074    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b492c\" (UniqueName: \"kubernetes.io/projected/e916f9c5-3853-42e2-a994-29ab114db2ff-kube-api-access-b492c\") pod \"payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk\" (UID: \"e916f9c5-3853-42e2-a994-29ab114db2ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:37:39.891310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.891235    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e916f9c5-3853-42e2-a994-29ab114db2ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk\" (UID: \"e916f9c5-3853-42e2-a994-29ab114db2ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:37:39.992329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.992290    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e916f9c5-3853-42e2-a994-29ab114db2ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk\" (UID: \"e916f9c5-3853-42e2-a994-29ab114db2ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:37:39.992507 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.992487    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b492c\" (UniqueName: \"kubernetes.io/projected/e916f9c5-3853-42e2-a994-29ab114db2ff-kube-api-access-b492c\") pod \"payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk\" (UID: \"e916f9c5-3853-42e2-a994-29ab114db2ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:37:39.993208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:39.993181    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e916f9c5-3853-42e2-a994-29ab114db2ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk\" (UID: \"e916f9c5-3853-42e2-a994-29ab114db2ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:37:40.056672 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:40.056590    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b492c\" (UniqueName: \"kubernetes.io/projected/e916f9c5-3853-42e2-a994-29ab114db2ff-kube-api-access-b492c\") pod \"payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk\" (UID: \"e916f9c5-3853-42e2-a994-29ab114db2ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:37:40.131188 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:40.131120    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:37:40.515093 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:40.515063    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"]
Nov 03 07:37:40.515572 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:37:40.515542    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode916f9c5_3853_42e2_a994_29ab114db2ff.slice/crio-c4c71b4c4e642ad5f545c9e10c74def122fb65b6e12ac102b30d81067a093aaf WatchSource:0}: Error finding container c4c71b4c4e642ad5f545c9e10c74def122fb65b6e12ac102b30d81067a093aaf: Status 404 returned error can't find the container with id c4c71b4c4e642ad5f545c9e10c74def122fb65b6e12ac102b30d81067a093aaf
Nov 03 07:37:40.573467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:40.573440    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk" event={"ID":"e916f9c5-3853-42e2-a994-29ab114db2ff","Type":"ContainerStarted","Data":"c4c71b4c4e642ad5f545c9e10c74def122fb65b6e12ac102b30d81067a093aaf"}
Nov 03 07:37:41.581879 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:41.581838    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk" event={"ID":"e916f9c5-3853-42e2-a994-29ab114db2ff","Type":"ContainerStarted","Data":"c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5"}
Nov 03 07:37:41.582320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:41.581924    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:37:41.703569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:41.703515    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk" podStartSLOduration=2.703501002 podStartE2EDuration="2.703501002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:41.702623903 +0000 UTC m=+2314.034992921" watchObservedRunningTime="2026-11-03 07:37:41.703501002 +0000 UTC m=+2314.035869996"
Nov 03 07:37:49.717296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.717263    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"]
Nov 03 07:37:49.721462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.721441    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:49.780684 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.780652    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"]
Nov 03 07:37:49.827494 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.827465    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:37:49.827700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.827682    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:37:49.827767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.827711    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:37:49.827767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.827734    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:37:49.836814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.836784    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rbx\" (UniqueName: \"kubernetes.io/projected/d8c01ba8-d733-4571-8faf-e83f306f9eee-kube-api-access-w9rbx\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:49.836952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.836875    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:49.837117 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.837098    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:49.938592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.938552    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rbx\" (UniqueName: \"kubernetes.io/projected/d8c01ba8-d733-4571-8faf-e83f306f9eee-kube-api-access-w9rbx\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:49.938798 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.938601    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:49.939010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.938986    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:49.939424 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.939396    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:49.941962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:49.941939    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:50.002365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:50.002293    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rbx\" (UniqueName: \"kubernetes.io/projected/d8c01ba8-d733-4571-8faf-e83f306f9eee-kube-api-access-w9rbx\") pod \"maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:50.035223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:50.035193    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:50.416510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:50.416478    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"]
Nov 03 07:37:50.417181 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:37:50.417155    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c01ba8_d733_4571_8faf_e83f306f9eee.slice/crio-594bc7ce76e99876013b216e23bd028e10d224f97cc5afcf3afe421bf48ea3b9 WatchSource:0}: Error finding container 594bc7ce76e99876013b216e23bd028e10d224f97cc5afcf3afe421bf48ea3b9: Status 404 returned error can't find the container with id 594bc7ce76e99876013b216e23bd028e10d224f97cc5afcf3afe421bf48ea3b9
Nov 03 07:37:50.656932 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:50.656896    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl" event={"ID":"d8c01ba8-d733-4571-8faf-e83f306f9eee","Type":"ContainerStarted","Data":"594bc7ce76e99876013b216e23bd028e10d224f97cc5afcf3afe421bf48ea3b9"}
Nov 03 07:37:51.665660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:51.665629    4470 generic.go:356] "Generic (PLEG): container finished" podID="d8c01ba8-d733-4571-8faf-e83f306f9eee" containerID="e7644fe603f8ab81e21fea246cac15a6f13a9ad070704f51b08fd7d80d1aba3f" exitCode=0
Nov 03 07:37:51.666061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:51.665707    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl" event={"ID":"d8c01ba8-d733-4571-8faf-e83f306f9eee","Type":"ContainerDied","Data":"e7644fe603f8ab81e21fea246cac15a6f13a9ad070704f51b08fd7d80d1aba3f"}
Nov 03 07:37:52.094940 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.094905    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"]
Nov 03 07:37:52.099209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.099186    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.197732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.197698    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"]
Nov 03 07:37:52.274247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.274191    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.274247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.274243    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68028204-1304-4646-8478-6a418f7c0144-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.274535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.274356    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.274620 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.274529    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.274620 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.274578    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.274761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.274620    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.274761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.274684    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m88c\" (UniqueName: \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-kube-api-access-5m88c\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.274761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.274736    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68028204-1304-4646-8478-6a418f7c0144-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.274907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.274839    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376047    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376091    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376117    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376167    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m88c\" (UniqueName: \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-kube-api-access-5m88c\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376197    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68028204-1304-4646-8478-6a418f7c0144-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376265    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376386    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376646 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376551    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376646 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376575    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376659    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376692    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68028204-1304-4646-8478-6a418f7c0144-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.376824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.376747    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.377068 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.377044    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68028204-1304-4646-8478-6a418f7c0144-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.377196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.377075    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.378621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.378597    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.379035 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.379013    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68028204-1304-4646-8478-6a418f7c0144-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.440967 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.440926    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.441148 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.441020    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m88c\" (UniqueName: \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-kube-api-access-5m88c\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.596607 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.596578    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:37:52.713320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.713286    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:52.823570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.823542    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:52.986356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.986248    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-xks-ca\") pod \"d8c01ba8-d733-4571-8faf-e83f306f9eee\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") "
Nov 03 07:37:52.986547 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.986353    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c01ba8-d733-4571-8faf-e83f306f9eee-kube-api-access-w9rbx\" (UniqueName: \"kubernetes.io/projected/d8c01ba8-d733-4571-8faf-e83f306f9eee-kube-api-access-w9rbx\") pod \"d8c01ba8-d733-4571-8faf-e83f306f9eee\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") "
Nov 03 07:37:52.986547 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.986399    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-openshift-ca\") pod \"d8c01ba8-d733-4571-8faf-e83f306f9eee\" (UID: \"d8c01ba8-d733-4571-8faf-e83f306f9eee\") "
Nov 03 07:37:52.987115 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.987079    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-openshift-ca" pod "d8c01ba8-d733-4571-8faf-e83f306f9eee" (UID: "d8c01ba8-d733-4571-8faf-e83f306f9eee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:37:52.989230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.989202    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-xks-ca" pod "d8c01ba8-d733-4571-8faf-e83f306f9eee" (UID: "d8c01ba8-d733-4571-8faf-e83f306f9eee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:37:52.989356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:52.989334    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c01ba8-d733-4571-8faf-e83f306f9eee-kube-api-access-w9rbx" pod "d8c01ba8-d733-4571-8faf-e83f306f9eee" (UID: "d8c01ba8-d733-4571-8faf-e83f306f9eee"). InnerVolumeSpecName "kube-api-access-w9rbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:37:53.088300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.088268    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:53.088300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.088296    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rbx\" (UniqueName: \"kubernetes.io/projected/d8c01ba8-d733-4571-8faf-e83f306f9eee-kube-api-access-w9rbx\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:53.088300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.088307    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c01ba8-d733-4571-8faf-e83f306f9eee-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:37:53.171660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.171627    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"]
Nov 03 07:37:53.172370 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:37:53.172342    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68028204_1304_4646_8478_6a418f7c0144.slice/crio-60f11b43a74406faab2c78033fd0ec25742574ffa6842a91fdfd96f67e2a4500 WatchSource:0}: Error finding container 60f11b43a74406faab2c78033fd0ec25742574ffa6842a91fdfd96f67e2a4500: Status 404 returned error can't find the container with id 60f11b43a74406faab2c78033fd0ec25742574ffa6842a91fdfd96f67e2a4500
Nov 03 07:37:53.178644 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.178608    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:37:53.178744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.178694    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:37:53.178744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.178724    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:37:53.681729 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.681684    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj" event={"ID":"68028204-1304-4646-8478-6a418f7c0144","Type":"ContainerStarted","Data":"f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644"}
Nov 03 07:37:53.681911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.681745    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj" event={"ID":"68028204-1304-4646-8478-6a418f7c0144","Type":"ContainerStarted","Data":"60f11b43a74406faab2c78033fd0ec25742574ffa6842a91fdfd96f67e2a4500"}
Nov 03 07:37:53.683277 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.683245    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"
Nov 03 07:37:53.683399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.683280    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl" event={"ID":"d8c01ba8-d733-4571-8faf-e83f306f9eee","Type":"ContainerDied","Data":"594bc7ce76e99876013b216e23bd028e10d224f97cc5afcf3afe421bf48ea3b9"}
Nov 03 07:37:53.683399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.683317    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="594bc7ce76e99876013b216e23bd028e10d224f97cc5afcf3afe421bf48ea3b9"
Nov 03 07:37:53.716555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.716521    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:53.807524 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:53.807450    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj" podStartSLOduration=1.80743137 podStartE2EDuration="1.80743137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:53.806505104 +0000 UTC m=+2326.138874098" watchObservedRunningTime="2026-11-03 07:37:53.80743137 +0000 UTC m=+2326.139800366"
Nov 03 07:37:54.715996 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:54.715957    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 07:37:54.716246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:54.716029    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj" podUID="68028204-1304-4646-8478-6a418f7c0144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 07:37:55.713955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:55.713917    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 07:37:55.714411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:55.713980    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj" podUID="68028204-1304-4646-8478-6a418f7c0144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 07:37:56.714527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:56.714496    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 07:37:56.714938 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:56.714547    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj" podUID="68028204-1304-4646-8478-6a418f7c0144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 07:37:57.713677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:57.713643    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 07:37:57.713836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:57.713694    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj" podUID="68028204-1304-4646-8478-6a418f7c0144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 07:37:58.714499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:58.714462    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 07:37:58.714907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:58.714526    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj" podUID="68028204-1304-4646-8478-6a418f7c0144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 07:37:59.248846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:59.248814    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"]
Nov 03 07:37:59.299639 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:59.299606    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5bdb10-pd4n4"]
Nov 03 07:37:59.718663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:59.718625    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:59.719168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:59.719120    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:59.720115 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:59.720095    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:37:59.808014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:37:59.807983    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545e8729-76b0-473c-8bbf-2e651e972d52" path="/var/lib/kubelet/pods/545e8729-76b0-473c-8bbf-2e651e972d52/volumes"
Nov 03 07:38:04.760572 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:38:04.760532    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:38:07.077251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.077213    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"]
Nov 03 07:38:07.077701 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.077409    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy" containerID="cri-o://5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2" gracePeriod=30
Nov 03 07:38:07.249944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.249908    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"]
Nov 03 07:38:07.252360 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.252326    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d8c01ba8-d733-4571-8faf-e83f306f9eee" containerName="revoke-keys"
Nov 03 07:38:07.252511 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.252499    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c01ba8-d733-4571-8faf-e83f306f9eee" containerName="revoke-keys"
Nov 03 07:38:07.252818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.252800    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c01ba8-d733-4571-8faf-e83f306f9eee" containerName="revoke-keys"
Nov 03 07:38:07.258810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.258791    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.295983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.295950    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"]
Nov 03 07:38:07.350331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.350099    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.350331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.350328    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.350651 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.350361    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.350651 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.350398    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee20c586-082c-4333-be74-5e8d49f84711-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.350802 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.350665    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.350802 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.350748    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee20c586-082c-4333-be74-5e8d49f84711-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.350912 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.350863    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46mqb\" (UniqueName: \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-kube-api-access-46mqb\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.353226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.350973    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.353226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.351050    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.451983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.451947    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.451983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.451984    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.452317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.452015    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee20c586-082c-4333-be74-5e8d49f84711-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.452317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.452045    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.452317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.452087    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee20c586-082c-4333-be74-5e8d49f84711-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.452317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.452157    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46mqb\" (UniqueName: \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-kube-api-access-46mqb\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.452317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.452234    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.452317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.452281    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.452734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.452325    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.452808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.452785    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.452898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.452868    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee20c586-082c-4333-be74-5e8d49f84711-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.453091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.453072    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.453360 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.453338    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.453585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.453566    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.455016 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.454990    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.456508 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.456489    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee20c586-082c-4333-be74-5e8d49f84711-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.517334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.517303    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.517503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.517397    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46mqb\" (UniqueName: \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-kube-api-access-46mqb\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.582724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.582692    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:07.613975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.613895    4470 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:38:07.614151 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:07.613978    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:38:08.026222 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.026195    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"]
Nov 03 07:38:08.026473 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:38:08.026451    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee20c586_082c_4333_be74_5e8d49f84711.slice/crio-4413069d0cab1815b2edf81b5e0f9ad6595a6ace74988d9efbc5af5acd6c8686 WatchSource:0}: Error finding container 4413069d0cab1815b2edf81b5e0f9ad6595a6ace74988d9efbc5af5acd6c8686: Status 404 returned error can't find the container with id 4413069d0cab1815b2edf81b5e0f9ad6595a6ace74988d9efbc5af5acd6c8686
Nov 03 07:38:08.029361 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.029321    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:38:08.029456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.029392    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:38:08.029456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.029428    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:38:08.069919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.069886    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"]
Nov 03 07:38:08.075708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.075687    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:08.131740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.131509    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:38:08.131740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.131579    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:38:08.131740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.131688    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:38:08.133591 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.133567    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:38:08.136410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.136359    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"]
Nov 03 07:38:08.169826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.169794    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:08.170005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.169860    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79brp\" (UniqueName: \"kubernetes.io/projected/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-kube-api-access-79brp\") pod \"maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:08.170076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.170040    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:08.271773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.271724    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:08.271990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.271810    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79brp\" (UniqueName: \"kubernetes.io/projected/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-kube-api-access-79brp\") pod \"maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:08.271990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.271914    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:08.272649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.272621    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:08.324218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.324182    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:38:08.334838 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.334806    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:08.335763 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.335735    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79brp\" (UniqueName: \"kubernetes.io/projected/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-kube-api-access-79brp\") pod \"maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:08.443452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.443425    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:38:08.451561 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.451532    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:08.806191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.806070    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9" event={"ID":"ee20c586-082c-4333-be74-5e8d49f84711","Type":"ContainerStarted","Data":"5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e"}
Nov 03 07:38:08.806191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.806173    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9" event={"ID":"ee20c586-082c-4333-be74-5e8d49f84711","Type":"ContainerStarted","Data":"4413069d0cab1815b2edf81b5e0f9ad6595a6ace74988d9efbc5af5acd6c8686"}
Nov 03 07:38:08.870158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.864860    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"]
Nov 03 07:38:08.943732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:08.943692    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9" podStartSLOduration=1.943678521 podStartE2EDuration="1.943678521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:38:08.9402668 +0000 UTC m=+2341.272635794" watchObservedRunningTime="2026-11-03 07:38:08.943678521 +0000 UTC m=+2341.276047525"
Nov 03 07:38:09.583530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:09.583483    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:09.584852 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:09.584824    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 07:38:09.584994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:09.584883    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9" podUID="ee20c586-082c-4333-be74-5e8d49f84711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 07:38:09.814436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:09.814401    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km" event={"ID":"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4","Type":"ContainerStarted","Data":"32d78cd0cf634b947dc6c40b51f0de7268a3a6c4523db753883a85d851ad676f"}
Nov 03 07:38:09.814647 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:09.814460    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km" event={"ID":"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4","Type":"ContainerStarted","Data":"0ae8e3e3b02b43056bb1b2847e2bd3a6d3d7544a45240cb431c765b6afe331b7"}
Nov 03 07:38:09.938853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:09.938745    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km" podStartSLOduration=1.938732319 podStartE2EDuration="1.938732319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:38:09.936910345 +0000 UTC m=+2342.269279343" watchObservedRunningTime="2026-11-03 07:38:09.938732319 +0000 UTC m=+2342.271101313"
Nov 03 07:38:10.583401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:10.583369    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 07:38:10.583564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:10.583424    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9" podUID="ee20c586-082c-4333-be74-5e8d49f84711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 07:38:10.824184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:10.824152    4470 generic.go:356] "Generic (PLEG): container finished" podID="1e47105a-35f2-4d3e-a017-66c0bdcc6ee4" containerID="32d78cd0cf634b947dc6c40b51f0de7268a3a6c4523db753883a85d851ad676f" exitCode=0
Nov 03 07:38:10.824371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:10.824228    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km" event={"ID":"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4","Type":"ContainerDied","Data":"32d78cd0cf634b947dc6c40b51f0de7268a3a6c4523db753883a85d851ad676f"}
Nov 03 07:38:11.583714 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:11.583676    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 07:38:11.584119 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:11.583754    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9" podUID="ee20c586-082c-4333-be74-5e8d49f84711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 07:38:12.074990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.074961    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:12.236841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.235707    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-kube-api-access-79brp\" (UniqueName: \"kubernetes.io/projected/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-kube-api-access-79brp\") pod \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") "
Nov 03 07:38:12.236841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.235966    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-xks-ca\") pod \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") "
Nov 03 07:38:12.236841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.236003    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-openshift-ca\") pod \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\" (UID: \"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4\") "
Nov 03 07:38:12.237675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.237632    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-openshift-ca" pod "1e47105a-35f2-4d3e-a017-66c0bdcc6ee4" (UID: "1e47105a-35f2-4d3e-a017-66c0bdcc6ee4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:38:12.240719 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.240494    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-xks-ca" pod "1e47105a-35f2-4d3e-a017-66c0bdcc6ee4" (UID: "1e47105a-35f2-4d3e-a017-66c0bdcc6ee4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:38:12.240928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.240901    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-kube-api-access-79brp" pod "1e47105a-35f2-4d3e-a017-66c0bdcc6ee4" (UID: "1e47105a-35f2-4d3e-a017-66c0bdcc6ee4"). InnerVolumeSpecName "kube-api-access-79brp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:38:12.339284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.339242    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.339284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.339287    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.339545 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.339304    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79brp\" (UniqueName: \"kubernetes.io/projected/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4-kube-api-access-79brp\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.401555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.401525    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:38:12.541962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.541867    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-podinfo\") pod \"c3c5935f-9218-450b-a7d5-8fd56685d917\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") "
Nov 03 07:38:12.541962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.541924    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-token\" (UniqueName: \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-token\") pod \"c3c5935f-9218-450b-a7d5-8fd56685d917\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") "
Nov 03 07:38:12.542247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.542018    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-kube-api-access-jhrtw\" (UniqueName: \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-kube-api-access-jhrtw\") pod \"c3c5935f-9218-450b-a7d5-8fd56685d917\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") "
Nov 03 07:38:12.542247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.542168    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-credential-socket\") pod \"c3c5935f-9218-450b-a7d5-8fd56685d917\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") "
Nov 03 07:38:12.542247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.542202    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-socket\") pod \"c3c5935f-9218-450b-a7d5-8fd56685d917\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") "
Nov 03 07:38:12.542413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.542265    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c5935f-9218-450b-a7d5-8fd56685d917-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c5935f-9218-450b-a7d5-8fd56685d917-istiod-ca-cert\") pod \"c3c5935f-9218-450b-a7d5-8fd56685d917\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") "
Nov 03 07:38:12.542413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.542311    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-data\") pod \"c3c5935f-9218-450b-a7d5-8fd56685d917\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") "
Nov 03 07:38:12.542413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.542355    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-certs\") pod \"c3c5935f-9218-450b-a7d5-8fd56685d917\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") "
Nov 03 07:38:12.542574 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.542464    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-envoy\") pod \"c3c5935f-9218-450b-a7d5-8fd56685d917\" (UID: \"c3c5935f-9218-450b-a7d5-8fd56685d917\") "
Nov 03 07:38:12.542574 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.542518    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-credential-socket" pod "c3c5935f-9218-450b-a7d5-8fd56685d917" (UID: "c3c5935f-9218-450b-a7d5-8fd56685d917"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:12.542964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.542938    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-socket" pod "c3c5935f-9218-450b-a7d5-8fd56685d917" (UID: "c3c5935f-9218-450b-a7d5-8fd56685d917"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:12.543127 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.543066    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.543307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.543285    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"]
Nov 03 07:38:12.543401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.543375    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-data" pod "c3c5935f-9218-450b-a7d5-8fd56685d917" (UID: "c3c5935f-9218-450b-a7d5-8fd56685d917"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:12.543441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.543424    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-certs" pod "c3c5935f-9218-450b-a7d5-8fd56685d917" (UID: "c3c5935f-9218-450b-a7d5-8fd56685d917"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:12.543478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.543430    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c5935f-9218-450b-a7d5-8fd56685d917-istiod-ca-cert" pod "c3c5935f-9218-450b-a7d5-8fd56685d917" (UID: "c3c5935f-9218-450b-a7d5-8fd56685d917"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:38:12.546104 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.546069    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-envoy" pod "c3c5935f-9218-450b-a7d5-8fd56685d917" (UID: "c3c5935f-9218-450b-a7d5-8fd56685d917"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:12.561228 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.546254    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-token" pod "c3c5935f-9218-450b-a7d5-8fd56685d917" (UID: "c3c5935f-9218-450b-a7d5-8fd56685d917"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:38:12.561228 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.546289    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-kube-api-access-jhrtw" pod "c3c5935f-9218-450b-a7d5-8fd56685d917" (UID: "c3c5935f-9218-450b-a7d5-8fd56685d917"). InnerVolumeSpecName "kube-api-access-jhrtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:38:12.573160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.570275    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-podinfo" pod "c3c5935f-9218-450b-a7d5-8fd56685d917" (UID: "c3c5935f-9218-450b-a7d5-8fd56685d917"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:38:12.583836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.583804    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 07:38:12.584205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.583863    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9" podUID="ee20c586-082c-4333-be74-5e8d49f84711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 07:38:12.596750 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.596720    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"]
Nov 03 07:38:12.600863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.600571    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"]
Nov 03 07:38:12.605622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.605591    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"]
Nov 03 07:38:12.645804 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.645749    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhrtw\" (UniqueName: \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-kube-api-access-jhrtw\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.645988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.645814    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.645988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.645832    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c5935f-9218-450b-a7d5-8fd56685d917-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.645988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.645844    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.645988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.645875    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.645988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.645887    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.645988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.645898    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.645988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.645910    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c5935f-9218-450b-a7d5-8fd56685d917-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:12.842154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.840906    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"
Nov 03 07:38:12.842154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.840931    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km" event={"ID":"1e47105a-35f2-4d3e-a017-66c0bdcc6ee4","Type":"ContainerDied","Data":"0ae8e3e3b02b43056bb1b2847e2bd3a6d3d7544a45240cb431c765b6afe331b7"}
Nov 03 07:38:12.842154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.840988    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ae8e3e3b02b43056bb1b2847e2bd3a6d3d7544a45240cb431c765b6afe331b7"
Nov 03 07:38:12.847152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.842996    4470 generic.go:356] "Generic (PLEG): container finished" podID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerID="5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2" exitCode=0
Nov 03 07:38:12.847152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.843099    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" event={"ID":"c3c5935f-9218-450b-a7d5-8fd56685d917","Type":"ContainerDied","Data":"5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2"}
Nov 03 07:38:12.847152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.843165    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9" event={"ID":"c3c5935f-9218-450b-a7d5-8fd56685d917","Type":"ContainerDied","Data":"9c997d18a6a76579ff530f7c10df24bfea89c67373fd26f68bd82dd086bb0543"}
Nov 03 07:38:12.847152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.843189    4470 scope.go:122] "RemoveContainer" containerID="5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2"
Nov 03 07:38:12.847152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.843304    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"
Nov 03 07:38:12.891995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.891657    4470 scope.go:122] "RemoveContainer" containerID="e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba"
Nov 03 07:38:12.904212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.904188    4470 scope.go:122] "RemoveContainer" containerID="5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2"
Nov 03 07:38:12.904552 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:38:12.904518    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2\": container with ID starting with 5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2 not found: ID does not exist" containerID="5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2"
Nov 03 07:38:12.904609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.904568    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2"} err="failed to get container status \"5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2\": rpc error: code = NotFound desc = could not find container \"5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2\": container with ID starting with 5dfc0d3eb9ca50fdc43ce8af7d05864cdc4b67b83c599cb11e317cf82e7254f2 not found: ID does not exist"
Nov 03 07:38:12.904609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.904594    4470 scope.go:122] "RemoveContainer" containerID="e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba"
Nov 03 07:38:12.904843 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:38:12.904821    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba\": container with ID starting with e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba not found: ID does not exist" containerID="e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba"
Nov 03 07:38:12.904899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:12.904852    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba"} err="failed to get container status \"e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba\": rpc error: code = NotFound desc = could not find container \"e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba\": container with ID starting with e68960a26b7d4437e47abd15e7a43dfca84582059a524fc3be0eb727ac9ebbba not found: ID does not exist"
Nov 03 07:38:13.234497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:13.234406    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"]
Nov 03 07:38:13.242693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:13.242649    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9"]
Nov 03 07:38:13.583119 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:13.583087    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 07:38:13.583416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:13.583382    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9" podUID="ee20c586-082c-4333-be74-5e8d49f84711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 07:38:13.807683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:13.807654    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" path="/var/lib/kubelet/pods/c3c5935f-9218-450b-a7d5-8fd56685d917/volumes"
Nov 03 07:38:14.117432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:14.117401    4470 scope.go:122] "RemoveContainer" containerID="8d14a6c535542d16aa328423b3f11676a287ad5e47a89dc5dac05963ddfab466"
Nov 03 07:38:14.128801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:14.128781    4470 scope.go:122] "RemoveContainer" containerID="c311e571f461b4ea040b5b26005667f0b445b3466bb08ed9db6c4b3d37d15a1b"
Nov 03 07:38:14.631748 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:14.631704    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:14.632235 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:14.632202    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:14.632919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:14.632904    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:38:23.484471 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:23.484436    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"]
Nov 03 07:38:23.594504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:23.594465    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"]
Nov 03 07:38:27.593620 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:27.593567    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" containerID="cri-o://c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50" gracePeriod=45
Nov 03 07:38:27.634532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:27.634433    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" containerID="cri-o://f9074f12e5b66412c6d04e939ee1e425929bb2093b4322bcd7a8f0014b52b654" gracePeriod=45
Nov 03 07:38:27.634742 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:27.634683    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" containerID="cri-o://340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372" gracePeriod=45
Nov 03 07:38:27.640378 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:27.640334    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" containerID="cri-o://b3c342f84e1e0b9059ebbbd75f2e6251fa4250bae1a2a56d12c5b3cc5d48676f" gracePeriod=45
Nov 03 07:38:27.809496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:27.809115    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": dial tcp 10.134.0.86:8000: connect: connection refused"
Nov 03 07:38:27.827256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:27.827026    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused"
Nov 03 07:38:27.836680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:27.836640    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused"
Nov 03 07:38:27.843721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:27.843479    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": dial tcp 10.134.0.83:8000: connect: connection refused"
Nov 03 07:38:27.989842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:27.989819    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:38:28.002172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.001121    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:38:28.002172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.001373    4470 generic.go:356] "Generic (PLEG): container finished" podID="62350718-e181-41ff-83b1-4d05a66539a7" containerID="c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50" exitCode=0
Nov 03 07:38:28.002172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.001423    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"
Nov 03 07:38:28.002172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.001603    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" event={"ID":"62350718-e181-41ff-83b1-4d05a66539a7","Type":"ContainerDied","Data":"c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50"}
Nov 03 07:38:28.002172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.001639    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn" event={"ID":"62350718-e181-41ff-83b1-4d05a66539a7","Type":"ContainerDied","Data":"81fb170e026f0f8b05690ac67728bc04286fdb20379cff2b27c632e39d8c8677"}
Nov 03 07:38:28.002172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.001658    4470 scope.go:122] "RemoveContainer" containerID="c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50"
Nov 03 07:38:28.003504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.003475    4470 generic.go:356] "Generic (PLEG): container finished" podID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerID="340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372" exitCode=0
Nov 03 07:38:28.003631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.003552    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" event={"ID":"bf4c497a-71a5-4cf8-866f-7630b8961efa","Type":"ContainerDied","Data":"340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372"}
Nov 03 07:38:28.003631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.003581    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2" event={"ID":"bf4c497a-71a5-4cf8-866f-7630b8961efa","Type":"ContainerDied","Data":"6059e1e08cf57255294c1ed8c380c63ff1ed0d5279885f8cd3cbfdcd4f5995c5"}
Nov 03 07:38:28.003752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.003634    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"
Nov 03 07:38:28.004596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.004546    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:38:28.006067 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.005231    4470 generic.go:356] "Generic (PLEG): container finished" podID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerID="b3c342f84e1e0b9059ebbbd75f2e6251fa4250bae1a2a56d12c5b3cc5d48676f" exitCode=0
Nov 03 07:38:28.006067 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.005380    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" event={"ID":"bf127fad-1026-4c13-ad33-28a806e7ebca","Type":"ContainerDied","Data":"b3c342f84e1e0b9059ebbbd75f2e6251fa4250bae1a2a56d12c5b3cc5d48676f"}
Nov 03 07:38:28.008081 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.008023    4470 generic.go:356] "Generic (PLEG): container finished" podID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerID="f9074f12e5b66412c6d04e939ee1e425929bb2093b4322bcd7a8f0014b52b654" exitCode=0
Nov 03 07:38:28.008245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.008067    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"
Nov 03 07:38:28.008245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.008205    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq" event={"ID":"e293d8c4-9555-4b8e-bae6-75b88e66a2f4","Type":"ContainerDied","Data":"f9074f12e5b66412c6d04e939ee1e425929bb2093b4322bcd7a8f0014b52b654"}
Nov 03 07:38:28.016844 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.016822    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:38:28.017702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.017688    4470 scope.go:122] "RemoveContainer" containerID="c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50"
Nov 03 07:38:28.017936 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:38:28.017918    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50\": container with ID starting with c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50 not found: ID does not exist" containerID="c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50"
Nov 03 07:38:28.018004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.017950    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50"} err="failed to get container status \"c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50\": rpc error: code = NotFound desc = could not find container \"c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50\": container with ID starting with c8cabc2a4da8091623296f062ad08e65d6ca1e1ec2805bc0f620ef8926d01c50 not found: ID does not exist"
Nov 03 07:38:28.018004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.017975    4470 scope.go:122] "RemoveContainer" containerID="340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372"
Nov 03 07:38:28.027194 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.027177    4470 scope.go:122] "RemoveContainer" containerID="340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372"
Nov 03 07:38:28.027485 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:38:28.027466    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372\": container with ID starting with 340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372 not found: ID does not exist" containerID="340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372"
Nov 03 07:38:28.027555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.027492    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372"} err="failed to get container status \"340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372\": rpc error: code = NotFound desc = could not find container \"340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372\": container with ID starting with 340eef6425d54c0f9a9e51d9e7ff239a711d97d63507a0022140051e5164a372 not found: ID does not exist"
Nov 03 07:38:28.027555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.027508    4470 scope.go:122] "RemoveContainer" containerID="f9074f12e5b66412c6d04e939ee1e425929bb2093b4322bcd7a8f0014b52b654"
Nov 03 07:38:28.112686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.112596    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf127fad-1026-4c13-ad33-28a806e7ebca-kube-api-access-strr5\" (UniqueName: \"kubernetes.io/projected/bf127fad-1026-4c13-ad33-28a806e7ebca-kube-api-access-strr5\") pod \"bf127fad-1026-4c13-ad33-28a806e7ebca\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") "
Nov 03 07:38:28.112686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.112656    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-home\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-home\") pod \"bf4c497a-71a5-4cf8-866f-7630b8961efa\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") "
Nov 03 07:38:28.112686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.112686    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-dshm\") pod \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") "
Nov 03 07:38:28.112979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.112742    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62350718-e181-41ff-83b1-4d05a66539a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/62350718-e181-41ff-83b1-4d05a66539a7-tls-certs\") pod \"62350718-e181-41ff-83b1-4d05a66539a7\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") "
Nov 03 07:38:28.112979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.112849    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf4c497a-71a5-4cf8-866f-7630b8961efa-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf4c497a-71a5-4cf8-866f-7630b8961efa-tls-certs\") pod \"bf4c497a-71a5-4cf8-866f-7630b8961efa\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") "
Nov 03 07:38:28.112979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.112910    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-dshm\") pod \"bf127fad-1026-4c13-ad33-28a806e7ebca\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") "
Nov 03 07:38:28.112979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.112943    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4c497a-71a5-4cf8-866f-7630b8961efa-kube-api-access-pfgxh\" (UniqueName: \"kubernetes.io/projected/bf4c497a-71a5-4cf8-866f-7630b8961efa-kube-api-access-pfgxh\") pod \"bf4c497a-71a5-4cf8-866f-7630b8961efa\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") "
Nov 03 07:38:28.113248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.112986    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tmp-dir\") pod \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") "
Nov 03 07:38:28.113248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113028    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-tmp-dir\") pod \"bf4c497a-71a5-4cf8-866f-7630b8961efa\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") "
Nov 03 07:38:28.113248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113057    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-tmp-dir\") pod \"bf127fad-1026-4c13-ad33-28a806e7ebca\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") "
Nov 03 07:38:28.113248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113150    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-home\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-home\") pod \"62350718-e181-41ff-83b1-4d05a66539a7\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") "
Nov 03 07:38:28.113248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113187    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tls-certs\") pod \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113257    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-home\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-home\") pod \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113287    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62350718-e181-41ff-83b1-4d05a66539a7-kube-api-access-x4tpb\" (UniqueName: \"kubernetes.io/projected/62350718-e181-41ff-83b1-4d05a66539a7-kube-api-access-x4tpb\") pod \"62350718-e181-41ff-83b1-4d05a66539a7\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113322    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-model-cache\") pod \"bf4c497a-71a5-4cf8-866f-7630b8961efa\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113377    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-dshm\") pod \"bf4c497a-71a5-4cf8-866f-7630b8961efa\" (UID: \"bf4c497a-71a5-4cf8-866f-7630b8961efa\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113433    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-model-cache\") pod \"bf127fad-1026-4c13-ad33-28a806e7ebca\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113460    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf127fad-1026-4c13-ad33-28a806e7ebca-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf127fad-1026-4c13-ad33-28a806e7ebca-tls-certs\") pod \"bf127fad-1026-4c13-ad33-28a806e7ebca\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113496    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-dshm\") pod \"62350718-e181-41ff-83b1-4d05a66539a7\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113591    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-home\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-home\") pod \"bf127fad-1026-4c13-ad33-28a806e7ebca\" (UID: \"bf127fad-1026-4c13-ad33-28a806e7ebca\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113633    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-kube-api-access-llsfx\" (UniqueName: \"kubernetes.io/projected/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-kube-api-access-llsfx\") pod \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113664    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-model-cache\") pod \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\" (UID: \"e293d8c4-9555-4b8e-bae6-75b88e66a2f4\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113697    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-tmp-dir\") pod \"62350718-e181-41ff-83b1-4d05a66539a7\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.113740    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-model-cache\") pod \"62350718-e181-41ff-83b1-4d05a66539a7\" (UID: \"62350718-e181-41ff-83b1-4d05a66539a7\") "
Nov 03 07:38:28.114890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.114751    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-model-cache" pod "62350718-e181-41ff-83b1-4d05a66539a7" (UID: "62350718-e181-41ff-83b1-4d05a66539a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.119995 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.119411    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-home" pod "62350718-e181-41ff-83b1-4d05a66539a7" (UID: "62350718-e181-41ff-83b1-4d05a66539a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.123998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.123969    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tmp-dir" pod "e293d8c4-9555-4b8e-bae6-75b88e66a2f4" (UID: "e293d8c4-9555-4b8e-bae6-75b88e66a2f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.127183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.125424    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tls-certs" pod "e293d8c4-9555-4b8e-bae6-75b88e66a2f4" (UID: "e293d8c4-9555-4b8e-bae6-75b88e66a2f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:38:28.127183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.126172    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-model-cache" pod "e293d8c4-9555-4b8e-bae6-75b88e66a2f4" (UID: "e293d8c4-9555-4b8e-bae6-75b88e66a2f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.127183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.126414    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-tmp-dir" pod "62350718-e181-41ff-83b1-4d05a66539a7" (UID: "62350718-e181-41ff-83b1-4d05a66539a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.127183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.126835    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-dshm" pod "bf127fad-1026-4c13-ad33-28a806e7ebca" (UID: "bf127fad-1026-4c13-ad33-28a806e7ebca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.127974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.127948    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-model-cache" pod "bf4c497a-71a5-4cf8-866f-7630b8961efa" (UID: "bf4c497a-71a5-4cf8-866f-7630b8961efa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.128189 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.128167    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-home" pod "e293d8c4-9555-4b8e-bae6-75b88e66a2f4" (UID: "e293d8c4-9555-4b8e-bae6-75b88e66a2f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.129336 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.129306    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-tmp-dir" pod "bf4c497a-71a5-4cf8-866f-7630b8961efa" (UID: "bf4c497a-71a5-4cf8-866f-7630b8961efa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.129524 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.129501    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-tmp-dir" pod "bf127fad-1026-4c13-ad33-28a806e7ebca" (UID: "bf127fad-1026-4c13-ad33-28a806e7ebca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.129960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.129936    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-model-cache" pod "bf127fad-1026-4c13-ad33-28a806e7ebca" (UID: "bf127fad-1026-4c13-ad33-28a806e7ebca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.130237 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.130214    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-home" pod "bf127fad-1026-4c13-ad33-28a806e7ebca" (UID: "bf127fad-1026-4c13-ad33-28a806e7ebca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.131054 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.131005    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62350718-e181-41ff-83b1-4d05a66539a7-tls-certs" pod "62350718-e181-41ff-83b1-4d05a66539a7" (UID: "62350718-e181-41ff-83b1-4d05a66539a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:38:28.131232 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.131208    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf127fad-1026-4c13-ad33-28a806e7ebca-tls-certs" pod "bf127fad-1026-4c13-ad33-28a806e7ebca" (UID: "bf127fad-1026-4c13-ad33-28a806e7ebca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:38:28.131733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.131580    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-dshm" pod "62350718-e181-41ff-83b1-4d05a66539a7" (UID: "62350718-e181-41ff-83b1-4d05a66539a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.131733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.131581    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4c497a-71a5-4cf8-866f-7630b8961efa-kube-api-access-pfgxh" pod "bf4c497a-71a5-4cf8-866f-7630b8961efa" (UID: "bf4c497a-71a5-4cf8-866f-7630b8961efa"). InnerVolumeSpecName "kube-api-access-pfgxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:38:28.131733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.131681    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf127fad-1026-4c13-ad33-28a806e7ebca-kube-api-access-strr5" pod "bf127fad-1026-4c13-ad33-28a806e7ebca" (UID: "bf127fad-1026-4c13-ad33-28a806e7ebca"). InnerVolumeSpecName "kube-api-access-strr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:38:28.131991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.131774    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-kube-api-access-llsfx" pod "e293d8c4-9555-4b8e-bae6-75b88e66a2f4" (UID: "e293d8c4-9555-4b8e-bae6-75b88e66a2f4"). InnerVolumeSpecName "kube-api-access-llsfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:38:28.132245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.132218    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-home" pod "bf4c497a-71a5-4cf8-866f-7630b8961efa" (UID: "bf4c497a-71a5-4cf8-866f-7630b8961efa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.133688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.133662    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf4c497a-71a5-4cf8-866f-7630b8961efa-tls-certs" pod "bf4c497a-71a5-4cf8-866f-7630b8961efa" (UID: "bf4c497a-71a5-4cf8-866f-7630b8961efa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:38:28.133761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.133683    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-dshm" pod "bf4c497a-71a5-4cf8-866f-7630b8961efa" (UID: "bf4c497a-71a5-4cf8-866f-7630b8961efa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.134445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.134420    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-dshm" pod "e293d8c4-9555-4b8e-bae6-75b88e66a2f4" (UID: "e293d8c4-9555-4b8e-bae6-75b88e66a2f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:28.135399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.135369    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62350718-e181-41ff-83b1-4d05a66539a7-kube-api-access-x4tpb" pod "62350718-e181-41ff-83b1-4d05a66539a7" (UID: "62350718-e181-41ff-83b1-4d05a66539a7"). InnerVolumeSpecName "kube-api-access-x4tpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:38:28.216228 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216193    4470 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tls-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216228 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216224    4470 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-home\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216237    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4tpb\" (UniqueName: \"kubernetes.io/projected/62350718-e181-41ff-83b1-4d05a66539a7-kube-api-access-x4tpb\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216251    4470 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-model-cache\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216264    4470 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-dshm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216276    4470 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-model-cache\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216286    4470 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf127fad-1026-4c13-ad33-28a806e7ebca-tls-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216300    4470 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-dshm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216310    4470 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-home\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216322    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llsfx\" (UniqueName: \"kubernetes.io/projected/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-kube-api-access-llsfx\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216334    4470 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-model-cache\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216348    4470 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-tmp-dir\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216359    4470 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-model-cache\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216371    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-strr5\" (UniqueName: \"kubernetes.io/projected/bf127fad-1026-4c13-ad33-28a806e7ebca-kube-api-access-strr5\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216382    4470 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-home\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216394    4470 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-dshm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216404    4470 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62350718-e181-41ff-83b1-4d05a66539a7-tls-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216415    4470 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf4c497a-71a5-4cf8-866f-7630b8961efa-tls-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216426    4470 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-dshm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216439    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfgxh\" (UniqueName: \"kubernetes.io/projected/bf4c497a-71a5-4cf8-866f-7630b8961efa-kube-api-access-pfgxh\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216453    4470 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e293d8c4-9555-4b8e-bae6-75b88e66a2f4-tmp-dir\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216467    4470 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4c497a-71a5-4cf8-866f-7630b8961efa-tmp-dir\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.216479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216481    4470 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf127fad-1026-4c13-ad33-28a806e7ebca-tmp-dir\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.217465 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.216493    4470 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62350718-e181-41ff-83b1-4d05a66539a7-home\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:28.485427 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.485342    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l" podUID="c710b3c4-f1e8-4a84-bd87-a02c252b7e09" containerName="payload-processing" containerID="cri-o://dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78" gracePeriod=25
Nov 03 07:38:28.600264 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.600214    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r" podUID="66f47047-32fd-45de-9f31-f1df5fb713da" containerName="payload-pre-processing" containerID="cri-o://d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d" gracePeriod=25
Nov 03 07:38:28.672598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.672541    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"]
Nov 03 07:38:28.679086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.679041    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/premium-w5-3fea3a-kserve-568f4956b8-rc6wq"]
Nov 03 07:38:28.789439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.789403    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"]
Nov 03 07:38:28.795507 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.794497    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-2-w5-3fea3a-kserve-6fc774dbd4-sfkdn"]
Nov 03 07:38:28.840958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.840916    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:38:28.893040 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.893012    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:38:28.909667 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.909639    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"]
Nov 03 07:38:28.912638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:28.912618    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/simulator-w5-3fea3a-kserve-56d768978f-68gs2"]
Nov 03 07:38:29.016196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.016109    4470 generic.go:356] "Generic (PLEG): container finished" podID="66f47047-32fd-45de-9f31-f1df5fb713da" containerID="d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d" exitCode=0
Nov 03 07:38:29.016196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.016165    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r" event={"ID":"66f47047-32fd-45de-9f31-f1df5fb713da","Type":"ContainerDied","Data":"d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d"}
Nov 03 07:38:29.016196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.016193    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"
Nov 03 07:38:29.016475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.016218    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r" event={"ID":"66f47047-32fd-45de-9f31-f1df5fb713da","Type":"ContainerDied","Data":"8f279c28943f533f29c786d9a8843f897a17fe08eaf6aa744dffb8852953ed90"}
Nov 03 07:38:29.016475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.016241    4470 scope.go:122] "RemoveContainer" containerID="d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d"
Nov 03 07:38:29.017854 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.017828    4470 generic.go:356] "Generic (PLEG): container finished" podID="c710b3c4-f1e8-4a84-bd87-a02c252b7e09" containerID="dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78" exitCode=0
Nov 03 07:38:29.017972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.017868    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"
Nov 03 07:38:29.017972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.017906    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l" event={"ID":"c710b3c4-f1e8-4a84-bd87-a02c252b7e09","Type":"ContainerDied","Data":"dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78"}
Nov 03 07:38:29.017972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.017960    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l" event={"ID":"c710b3c4-f1e8-4a84-bd87-a02c252b7e09","Type":"ContainerDied","Data":"b4282ecba618b18ce7ee0a9bedc728a0ccf9658ded1e6a76bfbbe188b0b5c569"}
Nov 03 07:38:29.020239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.020219    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"
Nov 03 07:38:29.020358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.020247    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9" event={"ID":"bf127fad-1026-4c13-ad33-28a806e7ebca","Type":"ContainerDied","Data":"1da32daddb2a7ad9373c7b128c182b1d5acc704f03c3df0e5f914158ed05aad8"}
Nov 03 07:38:29.030754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.030741    4470 scope.go:122] "RemoveContainer" containerID="d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d"
Nov 03 07:38:29.032739 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:38:29.032714    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d\": container with ID starting with d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d not found: ID does not exist" containerID="d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d"
Nov 03 07:38:29.032846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.032751    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d"} err="failed to get container status \"d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d\": rpc error: code = NotFound desc = could not find container \"d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d\": container with ID starting with d769907c5455ca232c5ddb7f945232b9ac36cc0eb0bad24c0e5f7ecf851b505d not found: ID does not exist"
Nov 03 07:38:29.032846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.032775    4470 scope.go:122] "RemoveContainer" containerID="dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78"
Nov 03 07:38:29.034474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.033367    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f47047-32fd-45de-9f31-f1df5fb713da-kube-api-access-7254m\" (UniqueName: \"kubernetes.io/projected/66f47047-32fd-45de-9f31-f1df5fb713da-kube-api-access-7254m\") pod \"66f47047-32fd-45de-9f31-f1df5fb713da\" (UID: \"66f47047-32fd-45de-9f31-f1df5fb713da\") "
Nov 03 07:38:29.034474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.033469    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f47047-32fd-45de-9f31-f1df5fb713da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f47047-32fd-45de-9f31-f1df5fb713da-plugins-config-volume\") pod \"66f47047-32fd-45de-9f31-f1df5fb713da\" (UID: \"66f47047-32fd-45de-9f31-f1df5fb713da\") "
Nov 03 07:38:29.034474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.033541    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-plugins-config-volume\") pod \"c710b3c4-f1e8-4a84-bd87-a02c252b7e09\" (UID: \"c710b3c4-f1e8-4a84-bd87-a02c252b7e09\") "
Nov 03 07:38:29.034474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.033646    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-kube-api-access-xdjpz\" (UniqueName: \"kubernetes.io/projected/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-kube-api-access-xdjpz\") pod \"c710b3c4-f1e8-4a84-bd87-a02c252b7e09\" (UID: \"c710b3c4-f1e8-4a84-bd87-a02c252b7e09\") "
Nov 03 07:38:29.034474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.034327    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f47047-32fd-45de-9f31-f1df5fb713da-plugins-config-volume" pod "66f47047-32fd-45de-9f31-f1df5fb713da" (UID: "66f47047-32fd-45de-9f31-f1df5fb713da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:38:29.034474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.034385    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-plugins-config-volume" pod "c710b3c4-f1e8-4a84-bd87-a02c252b7e09" (UID: "c710b3c4-f1e8-4a84-bd87-a02c252b7e09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:38:29.036762 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.036733    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-kube-api-access-xdjpz" pod "c710b3c4-f1e8-4a84-bd87-a02c252b7e09" (UID: "c710b3c4-f1e8-4a84-bd87-a02c252b7e09"). InnerVolumeSpecName "kube-api-access-xdjpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:38:29.036957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.036934    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f47047-32fd-45de-9f31-f1df5fb713da-kube-api-access-7254m" pod "66f47047-32fd-45de-9f31-f1df5fb713da" (UID: "66f47047-32fd-45de-9f31-f1df5fb713da"). InnerVolumeSpecName "kube-api-access-7254m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:38:29.044550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.044531    4470 scope.go:122] "RemoveContainer" containerID="dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78"
Nov 03 07:38:29.044874 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:38:29.044854    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78\": container with ID starting with dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78 not found: ID does not exist" containerID="dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78"
Nov 03 07:38:29.044938 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.044883    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78"} err="failed to get container status \"dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78\": rpc error: code = NotFound desc = could not find container \"dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78\": container with ID starting with dd305a591e73dc7e7d0ca23690cc62c42794583c326bd1bb07de4530da1e0e78 not found: ID does not exist"
Nov 03 07:38:29.044938 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.044900    4470 scope.go:122] "RemoveContainer" containerID="b3c342f84e1e0b9059ebbbd75f2e6251fa4250bae1a2a56d12c5b3cc5d48676f"
Nov 03 07:38:29.135272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.135245    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7254m\" (UniqueName: \"kubernetes.io/projected/66f47047-32fd-45de-9f31-f1df5fb713da-kube-api-access-7254m\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:29.135272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.135271    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f47047-32fd-45de-9f31-f1df5fb713da-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:29.135478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.135282    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:29.135478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.135291    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdjpz\" (UniqueName: \"kubernetes.io/projected/c710b3c4-f1e8-4a84-bd87-a02c252b7e09-kube-api-access-xdjpz\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:29.267201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.267093    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"]
Nov 03 07:38:29.272756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.272732    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3fea3a/distinct-w5-3fea3a-kserve-75d7fbcb99-tbmt9"]
Nov 03 07:38:29.496385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.496346    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"]
Nov 03 07:38:29.502790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.502761    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l"]
Nov 03 07:38:29.616964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.616937    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"]
Nov 03 07:38:29.622503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.622477    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r"]
Nov 03 07:38:29.808120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.808087    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62350718-e181-41ff-83b1-4d05a66539a7" path="/var/lib/kubelet/pods/62350718-e181-41ff-83b1-4d05a66539a7/volumes"
Nov 03 07:38:29.808552 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.808538    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f47047-32fd-45de-9f31-f1df5fb713da" path="/var/lib/kubelet/pods/66f47047-32fd-45de-9f31-f1df5fb713da/volumes"
Nov 03 07:38:29.808941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.808920    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" path="/var/lib/kubelet/pods/bf127fad-1026-4c13-ad33-28a806e7ebca/volumes"
Nov 03 07:38:29.809663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.809648    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" path="/var/lib/kubelet/pods/bf4c497a-71a5-4cf8-866f-7630b8961efa/volumes"
Nov 03 07:38:29.810017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.810004    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c710b3c4-f1e8-4a84-bd87-a02c252b7e09" path="/var/lib/kubelet/pods/c710b3c4-f1e8-4a84-bd87-a02c252b7e09/volumes"
Nov 03 07:38:29.810431 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:29.810418    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" path="/var/lib/kubelet/pods/e293d8c4-9555-4b8e-bae6-75b88e66a2f4/volumes"
Nov 03 07:38:31.279440 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:31.279398    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"]
Nov 03 07:38:31.279850 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:31.279675    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy" containerID="cri-o://ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a" gracePeriod=30
Nov 03 07:38:32.058331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:32.058295    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"]
Nov 03 07:38:32.061957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:32.061938    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf88ec81-cz6xn"]
Nov 03 07:38:33.806586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:33.806557    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="962feb66-3458-4d86-a7f7-690c4192c916" path="/var/lib/kubelet/pods/962feb66-3458-4d86-a7f7-690c4192c916/volumes"
Nov 03 07:38:34.761236 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:38:34.761200    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:38:36.552284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.552213    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:38:36.635334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635300    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-kube-api-access-5pzsb\" (UniqueName: \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-kube-api-access-5pzsb\") pod \"662121e8-340c-4638-9be4-de860e5cc823\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") "
Nov 03 07:38:36.635541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635350    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-socket\") pod \"662121e8-340c-4638-9be4-de860e5cc823\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") "
Nov 03 07:38:36.635541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635413    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-certs\") pod \"662121e8-340c-4638-9be4-de860e5cc823\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") "
Nov 03 07:38:36.635541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635462    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/662121e8-340c-4638-9be4-de860e5cc823-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662121e8-340c-4638-9be4-de860e5cc823-istio-podinfo\") pod \"662121e8-340c-4638-9be4-de860e5cc823\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") "
Nov 03 07:38:36.635541 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635510    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-data\") pod \"662121e8-340c-4638-9be4-de860e5cc823\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") "
Nov 03 07:38:36.635783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635541    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-credential-socket\") pod \"662121e8-340c-4638-9be4-de860e5cc823\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") "
Nov 03 07:38:36.635783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635627    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-istio-token\" (UniqueName: \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-istio-token\") pod \"662121e8-340c-4638-9be4-de860e5cc823\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") "
Nov 03 07:38:36.635783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635687    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662121e8-340c-4638-9be4-de860e5cc823-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662121e8-340c-4638-9be4-de860e5cc823-istiod-ca-cert\") pod \"662121e8-340c-4638-9be4-de860e5cc823\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") "
Nov 03 07:38:36.635783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635711    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-socket" pod "662121e8-340c-4638-9be4-de860e5cc823" (UID: "662121e8-340c-4638-9be4-de860e5cc823"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:36.635783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635763    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-certs" pod "662121e8-340c-4638-9be4-de860e5cc823" (UID: "662121e8-340c-4638-9be4-de860e5cc823"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:36.636040 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635789    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-envoy\") pod \"662121e8-340c-4638-9be4-de860e5cc823\" (UID: \"662121e8-340c-4638-9be4-de860e5cc823\") "
Nov 03 07:38:36.636040 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635796    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-data" pod "662121e8-340c-4638-9be4-de860e5cc823" (UID: "662121e8-340c-4638-9be4-de860e5cc823"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:36.636040 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.635959    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-credential-socket" pod "662121e8-340c-4638-9be4-de860e5cc823" (UID: "662121e8-340c-4638-9be4-de860e5cc823"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:36.636418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.636064    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662121e8-340c-4638-9be4-de860e5cc823-istiod-ca-cert" pod "662121e8-340c-4638-9be4-de860e5cc823" (UID: "662121e8-340c-4638-9be4-de860e5cc823"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:38:36.636418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.636377    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:36.636418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.636393    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:36.636418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.636401    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:36.636418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.636411    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662121e8-340c-4638-9be4-de860e5cc823-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:36.636418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.636419    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:36.637927 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.637896    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/662121e8-340c-4638-9be4-de860e5cc823-istio-podinfo" pod "662121e8-340c-4638-9be4-de860e5cc823" (UID: "662121e8-340c-4638-9be4-de860e5cc823"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:38:36.637927 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.637911    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-kube-api-access-5pzsb" pod "662121e8-340c-4638-9be4-de860e5cc823" (UID: "662121e8-340c-4638-9be4-de860e5cc823"). InnerVolumeSpecName "kube-api-access-5pzsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:38:36.638077 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.637925    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-istio-token" pod "662121e8-340c-4638-9be4-de860e5cc823" (UID: "662121e8-340c-4638-9be4-de860e5cc823"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:38:36.638321 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.638304    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-envoy" pod "662121e8-340c-4638-9be4-de860e5cc823" (UID: "662121e8-340c-4638-9be4-de860e5cc823"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:38:36.737205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.737176    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662121e8-340c-4638-9be4-de860e5cc823-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:36.737205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.737202    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:36.737205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.737213    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662121e8-340c-4638-9be4-de860e5cc823-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:36.737413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:36.737222    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pzsb\" (UniqueName: \"kubernetes.io/projected/662121e8-340c-4638-9be4-de860e5cc823-kube-api-access-5pzsb\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:38:37.079264 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:37.079234    4470 generic.go:356] "Generic (PLEG): container finished" podID="662121e8-340c-4638-9be4-de860e5cc823" containerID="ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a" exitCode=0
Nov 03 07:38:37.079460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:37.079286    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"
Nov 03 07:38:37.079460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:37.079322    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" event={"ID":"662121e8-340c-4638-9be4-de860e5cc823","Type":"ContainerDied","Data":"ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a"}
Nov 03 07:38:37.079460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:37.079378    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb" event={"ID":"662121e8-340c-4638-9be4-de860e5cc823","Type":"ContainerDied","Data":"3ecf9044d288e60eafd3a1ad45f1c090b81f73b6a340c7384a4411343e5d08a1"}
Nov 03 07:38:37.079460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:37.079401    4470 scope.go:122] "RemoveContainer" containerID="ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a"
Nov 03 07:38:37.090761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:37.090743    4470 scope.go:122] "RemoveContainer" containerID="ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a"
Nov 03 07:38:37.091021 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:38:37.091005    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a\": container with ID starting with ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a not found: ID does not exist" containerID="ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a"
Nov 03 07:38:37.091080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:37.091030    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a"} err="failed to get container status \"ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a\": rpc error: code = NotFound desc = could not find container \"ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a\": container with ID starting with ddfdc545ae435be7f4114c6d914901a42092a3c70ca8ad1659ae15b3e98ed35a not found: ID does not exist"
Nov 03 07:38:37.274895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:37.274857    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"]
Nov 03 07:38:37.312113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:37.312075    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb"]
Nov 03 07:38:37.806854 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:37.806822    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662121e8-340c-4638-9be4-de860e5cc823" path="/var/lib/kubelet/pods/662121e8-340c-4638-9be4-de860e5cc823/volumes"
Nov 03 07:38:52.498474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:52.498406    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"]
Nov 03 07:38:52.548805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:52.548777    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-295fba7c-mkbpl"]
Nov 03 07:38:53.806348 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:38:53.806315    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c01ba8-d733-4571-8faf-e83f306f9eee" path="/var/lib/kubelet/pods/d8c01ba8-d733-4571-8faf-e83f306f9eee/volumes"
Nov 03 07:39:04.760849 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:39:04.760808    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:39:06.908965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.908932    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"]
Nov 03 07:39:06.909575 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909559    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy"
Nov 03 07:39:06.909621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909577    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy"
Nov 03 07:39:06.909621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909589    4470 cpu_manager.go:414] "containerMap: removing container" podUID="1e47105a-35f2-4d3e-a017-66c0bdcc6ee4" containerName="revoke-keys"
Nov 03 07:39:06.909621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909595    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e47105a-35f2-4d3e-a017-66c0bdcc6ee4" containerName="revoke-keys"
Nov 03 07:39:06.909621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909605    4470 cpu_manager.go:414] "containerMap: removing container" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main"
Nov 03 07:39:06.909621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909610    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main"
Nov 03 07:39:06.909621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909619    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909623    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909633    4470 cpu_manager.go:414] "containerMap: removing container" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909637    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909643    4470 cpu_manager.go:414] "containerMap: removing container" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909647    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909659    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909664    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909676    4470 cpu_manager.go:414] "containerMap: removing container" podUID="66f47047-32fd-45de-9f31-f1df5fb713da" containerName="payload-pre-processing"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909680    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f47047-32fd-45de-9f31-f1df5fb713da" containerName="payload-pre-processing"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909690    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c710b3c4-f1e8-4a84-bd87-a02c252b7e09" containerName="payload-processing"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909694    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c710b3c4-f1e8-4a84-bd87-a02c252b7e09" containerName="payload-processing"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909713    4470 cpu_manager.go:414] "containerMap: removing container" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909717    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy"
Nov 03 07:39:06.909807 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909809    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="662121e8-340c-4638-9be4-de860e5cc823" containerName="istio-proxy"
Nov 03 07:39:06.910262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909819    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy"
Nov 03 07:39:06.910262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909826    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e47105a-35f2-4d3e-a017-66c0bdcc6ee4" containerName="revoke-keys"
Nov 03 07:39:06.910262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909832    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf127fad-1026-4c13-ad33-28a806e7ebca" containerName="main"
Nov 03 07:39:06.910262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909837    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e293d8c4-9555-4b8e-bae6-75b88e66a2f4" containerName="main"
Nov 03 07:39:06.910262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909844    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4c497a-71a5-4cf8-866f-7630b8961efa" containerName="main"
Nov 03 07:39:06.910262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909849    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="62350718-e181-41ff-83b1-4d05a66539a7" containerName="main"
Nov 03 07:39:06.910262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909857    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f47047-32fd-45de-9f31-f1df5fb713da" containerName="payload-pre-processing"
Nov 03 07:39:06.910262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.909861    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c710b3c4-f1e8-4a84-bd87-a02c252b7e09" containerName="payload-processing"
Nov 03 07:39:06.912945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.912923    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:06.968773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.968741    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:39:06.969011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.968825    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:39:06.972111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:06.972078    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"]
Nov 03 07:39:07.018344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.018311    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:39:07.018535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.018441    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:39:07.021615 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.021591    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"]
Nov 03 07:39:07.022269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.022245    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c5935f-9218-450b-a7d5-8fd56685d917" containerName="istio-proxy"
Nov 03 07:39:07.023171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.023128    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:07.023330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.023306    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:07.023473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.023456    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z94rz\" (UniqueName: \"kubernetes.io/projected/d477fb64-5ef6-4e61-977f-4477417704f6-kube-api-access-z94rz\") pod \"maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:07.025689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.025673    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:07.069948 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.069917    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"]
Nov 03 07:39:07.093076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.092992    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:07.108181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.108126    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"]
Nov 03 07:39:07.124503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.124464    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:07.124718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.124530    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-868fc96d-5gjv5\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:07.124718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.124674    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:07.124866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.124787    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z94rz\" (UniqueName: \"kubernetes.io/projected/d477fb64-5ef6-4e61-977f-4477417704f6-kube-api-access-z94rz\") pod \"maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:07.124866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.124824    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-868fc96d-5gjv5\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:07.124980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.124926    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmgc\" (UniqueName: \"kubernetes.io/projected/ad571469-adaa-4e64-b88a-6adeacfb6bd9-kube-api-access-tqmgc\") pod \"maas-api-revoke-sub-868fc96d-5gjv5\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:07.125211 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.125175    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:07.129706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.128390    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:07.134272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.134245    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"]
Nov 03 07:39:07.189112 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.189047    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z94rz\" (UniqueName: \"kubernetes.io/projected/d477fb64-5ef6-4e61-977f-4477417704f6-kube-api-access-z94rz\") pod \"maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:07.226405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.226374    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b90089-4287-45be-a343-b3ed069e0188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d93b019-2w86j\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:07.226597 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.226499    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-868fc96d-5gjv5\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:07.226597 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.226591    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b90089-4287-45be-a343-b3ed069e0188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d93b019-2w86j\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:07.226768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.226741    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-868fc96d-5gjv5\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:07.226839 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.226802    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htk7z\" (UniqueName: \"kubernetes.io/projected/40b90089-4287-45be-a343-b3ed069e0188-kube-api-access-htk7z\") pod \"maas-api-revoke-sub-9d93b019-2w86j\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:07.226889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.226875    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmgc\" (UniqueName: \"kubernetes.io/projected/ad571469-adaa-4e64-b88a-6adeacfb6bd9-kube-api-access-tqmgc\") pod \"maas-api-revoke-sub-868fc96d-5gjv5\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:07.227258 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.227235    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-868fc96d-5gjv5\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:07.228308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.228283    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:07.229283 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.229263    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-868fc96d-5gjv5\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:07.291747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.291688    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmgc\" (UniqueName: \"kubernetes.io/projected/ad571469-adaa-4e64-b88a-6adeacfb6bd9-kube-api-access-tqmgc\") pod \"maas-api-revoke-sub-868fc96d-5gjv5\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:07.329345 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.329312    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htk7z\" (UniqueName: \"kubernetes.io/projected/40b90089-4287-45be-a343-b3ed069e0188-kube-api-access-htk7z\") pod \"maas-api-revoke-sub-9d93b019-2w86j\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:07.329522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.329457    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b90089-4287-45be-a343-b3ed069e0188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d93b019-2w86j\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:07.329600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.329582    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b90089-4287-45be-a343-b3ed069e0188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d93b019-2w86j\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:07.330262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.330235    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b90089-4287-45be-a343-b3ed069e0188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d93b019-2w86j\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:07.335417 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.335387    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b90089-4287-45be-a343-b3ed069e0188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d93b019-2w86j\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:07.342311 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.342287    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:07.397792 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.397760    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htk7z\" (UniqueName: \"kubernetes.io/projected/40b90089-4287-45be-a343-b3ed069e0188-kube-api-access-htk7z\") pod \"maas-api-revoke-sub-9d93b019-2w86j\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:07.414022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.413999    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:07.615334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.615307    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"]
Nov 03 07:39:07.615986 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:39:07.615964    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd477fb64_5ef6_4e61_977f_4477417704f6.slice/crio-78c86eee3a53174b447a99b0549c3fe39ba9927b7bda787abe5d55a24ebfa73b WatchSource:0}: Error finding container 78c86eee3a53174b447a99b0549c3fe39ba9927b7bda787abe5d55a24ebfa73b: Status 404 returned error can't find the container with id 78c86eee3a53174b447a99b0549c3fe39ba9927b7bda787abe5d55a24ebfa73b
Nov 03 07:39:07.726810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.726784    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"]
Nov 03 07:39:07.727724 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:39:07.727702    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad571469_adaa_4e64_b88a_6adeacfb6bd9.slice/crio-6baedbd82bf5e5caea45f18da260cb32facfc4e1d6dbf464380c4fde3d55c2ee WatchSource:0}: Error finding container 6baedbd82bf5e5caea45f18da260cb32facfc4e1d6dbf464380c4fde3d55c2ee: Status 404 returned error can't find the container with id 6baedbd82bf5e5caea45f18da260cb32facfc4e1d6dbf464380c4fde3d55c2ee
Nov 03 07:39:07.816508 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:07.816479    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"]
Nov 03 07:39:07.817065 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:39:07.817040    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b90089_4287_45be_a343_b3ed069e0188.slice/crio-760801dc53a7f8b3d139b270d523c6195871a8ad627f2522c43bb38fdf93a326 WatchSource:0}: Error finding container 760801dc53a7f8b3d139b270d523c6195871a8ad627f2522c43bb38fdf93a326: Status 404 returned error can't find the container with id 760801dc53a7f8b3d139b270d523c6195871a8ad627f2522c43bb38fdf93a326
Nov 03 07:39:08.347318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:08.347271    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j" event={"ID":"40b90089-4287-45be-a343-b3ed069e0188","Type":"ContainerStarted","Data":"4a3458df700483a3ee8a683fc1df81ebad80536dd839a52db7cb02b58565ceb3"}
Nov 03 07:39:08.347772 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:08.347429    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j" event={"ID":"40b90089-4287-45be-a343-b3ed069e0188","Type":"ContainerStarted","Data":"760801dc53a7f8b3d139b270d523c6195871a8ad627f2522c43bb38fdf93a326"}
Nov 03 07:39:08.350016 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:08.349989    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5" event={"ID":"ad571469-adaa-4e64-b88a-6adeacfb6bd9","Type":"ContainerStarted","Data":"942f047431d1eb500330617c9f02cd604ba6c4aa4b37670afa0ab6644ae43415"}
Nov 03 07:39:08.350246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:08.350220    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5" event={"ID":"ad571469-adaa-4e64-b88a-6adeacfb6bd9","Type":"ContainerStarted","Data":"6baedbd82bf5e5caea45f18da260cb32facfc4e1d6dbf464380c4fde3d55c2ee"}
Nov 03 07:39:08.351786 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:08.351765    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn" event={"ID":"d477fb64-5ef6-4e61-977f-4477417704f6","Type":"ContainerStarted","Data":"ecb64939aa90edd50a4172c47c5605144e1b762ec3eea3af5ea1a780fe2efc8a"}
Nov 03 07:39:08.351897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:08.351801    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn" event={"ID":"d477fb64-5ef6-4e61-977f-4477417704f6","Type":"ContainerStarted","Data":"78c86eee3a53174b447a99b0549c3fe39ba9927b7bda787abe5d55a24ebfa73b"}
Nov 03 07:39:08.468344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:08.468294    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j" podStartSLOduration=1.468281657 podStartE2EDuration="1.468281657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:08.467213065 +0000 UTC m=+2400.799582059" watchObservedRunningTime="2026-11-03 07:39:08.468281657 +0000 UTC m=+2400.800650651"
Nov 03 07:39:08.586906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:08.586858    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5" podStartSLOduration=2.5868462819999998 podStartE2EDuration="2.586846282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:08.586510008 +0000 UTC m=+2400.918879002" watchObservedRunningTime="2026-11-03 07:39:08.586846282 +0000 UTC m=+2400.919215268"
Nov 03 07:39:08.708656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:08.708556    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn" podStartSLOduration=2.708537602 podStartE2EDuration="2.708537602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:08.706461869 +0000 UTC m=+2401.038830864" watchObservedRunningTime="2026-11-03 07:39:08.708537602 +0000 UTC m=+2401.040906595"
Nov 03 07:39:09.360885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:09.360850    4470 generic.go:356] "Generic (PLEG): container finished" podID="ad571469-adaa-4e64-b88a-6adeacfb6bd9" containerID="942f047431d1eb500330617c9f02cd604ba6c4aa4b37670afa0ab6644ae43415" exitCode=0
Nov 03 07:39:09.361366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:09.360935    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5" event={"ID":"ad571469-adaa-4e64-b88a-6adeacfb6bd9","Type":"ContainerDied","Data":"942f047431d1eb500330617c9f02cd604ba6c4aa4b37670afa0ab6644ae43415"}
Nov 03 07:39:09.362691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:09.362672    4470 generic.go:356] "Generic (PLEG): container finished" podID="d477fb64-5ef6-4e61-977f-4477417704f6" containerID="ecb64939aa90edd50a4172c47c5605144e1b762ec3eea3af5ea1a780fe2efc8a" exitCode=0
Nov 03 07:39:09.362845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:09.362750    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn" event={"ID":"d477fb64-5ef6-4e61-977f-4477417704f6","Type":"ContainerDied","Data":"ecb64939aa90edd50a4172c47c5605144e1b762ec3eea3af5ea1a780fe2efc8a"}
Nov 03 07:39:09.364274 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:09.364255    4470 generic.go:356] "Generic (PLEG): container finished" podID="40b90089-4287-45be-a343-b3ed069e0188" containerID="4a3458df700483a3ee8a683fc1df81ebad80536dd839a52db7cb02b58565ceb3" exitCode=0
Nov 03 07:39:09.364375 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:09.364278    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j" event={"ID":"40b90089-4287-45be-a343-b3ed069e0188","Type":"ContainerDied","Data":"4a3458df700483a3ee8a683fc1df81ebad80536dd839a52db7cb02b58565ceb3"}
Nov 03 07:39:10.564292 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.564271    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:10.595065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.595039    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:10.598414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.598396    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:10.677881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.677791    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-xks-ca\") pod \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") "
Nov 03 07:39:10.678069 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.677914    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad571469-adaa-4e64-b88a-6adeacfb6bd9-kube-api-access-tqmgc\" (UniqueName: \"kubernetes.io/projected/ad571469-adaa-4e64-b88a-6adeacfb6bd9-kube-api-access-tqmgc\") pod \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") "
Nov 03 07:39:10.678069 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.677963    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-openshift-ca\") pod \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\" (UID: \"ad571469-adaa-4e64-b88a-6adeacfb6bd9\") "
Nov 03 07:39:10.678405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.678371    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-openshift-ca" pod "ad571469-adaa-4e64-b88a-6adeacfb6bd9" (UID: "ad571469-adaa-4e64-b88a-6adeacfb6bd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:39:10.678553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.678540    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:39:10.680271 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.680247    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad571469-adaa-4e64-b88a-6adeacfb6bd9-kube-api-access-tqmgc" pod "ad571469-adaa-4e64-b88a-6adeacfb6bd9" (UID: "ad571469-adaa-4e64-b88a-6adeacfb6bd9"). InnerVolumeSpecName "kube-api-access-tqmgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:39:10.680377 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.680338    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-xks-ca" pod "ad571469-adaa-4e64-b88a-6adeacfb6bd9" (UID: "ad571469-adaa-4e64-b88a-6adeacfb6bd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:39:10.779344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.779305    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b90089-4287-45be-a343-b3ed069e0188-kube-api-access-htk7z\" (UniqueName: \"kubernetes.io/projected/40b90089-4287-45be-a343-b3ed069e0188-kube-api-access-htk7z\") pod \"40b90089-4287-45be-a343-b3ed069e0188\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") "
Nov 03 07:39:10.779500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.779452    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-xks-ca\") pod \"d477fb64-5ef6-4e61-977f-4477417704f6\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") "
Nov 03 07:39:10.779500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.779480    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b90089-4287-45be-a343-b3ed069e0188-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b90089-4287-45be-a343-b3ed069e0188-maas-api-openshift-ca\") pod \"40b90089-4287-45be-a343-b3ed069e0188\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") "
Nov 03 07:39:10.779627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.779526    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-openshift-ca\") pod \"d477fb64-5ef6-4e61-977f-4477417704f6\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") "
Nov 03 07:39:10.779627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.779551    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b90089-4287-45be-a343-b3ed069e0188-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b90089-4287-45be-a343-b3ed069e0188-maas-api-xks-ca\") pod \"40b90089-4287-45be-a343-b3ed069e0188\" (UID: \"40b90089-4287-45be-a343-b3ed069e0188\") "
Nov 03 07:39:10.779759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.779729    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d477fb64-5ef6-4e61-977f-4477417704f6-kube-api-access-z94rz\" (UniqueName: \"kubernetes.io/projected/d477fb64-5ef6-4e61-977f-4477417704f6-kube-api-access-z94rz\") pod \"d477fb64-5ef6-4e61-977f-4477417704f6\" (UID: \"d477fb64-5ef6-4e61-977f-4477417704f6\") "
Nov 03 07:39:10.779921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.779886    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-openshift-ca" pod "d477fb64-5ef6-4e61-977f-4477417704f6" (UID: "d477fb64-5ef6-4e61-977f-4477417704f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:39:10.780060 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.779886    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b90089-4287-45be-a343-b3ed069e0188-maas-api-openshift-ca" pod "40b90089-4287-45be-a343-b3ed069e0188" (UID: "40b90089-4287-45be-a343-b3ed069e0188"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:39:10.780387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.780360    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b90089-4287-45be-a343-b3ed069e0188-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:39:10.780504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.780391    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:39:10.780504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.780405    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad571469-adaa-4e64-b88a-6adeacfb6bd9-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:39:10.780504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.780417    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqmgc\" (UniqueName: \"kubernetes.io/projected/ad571469-adaa-4e64-b88a-6adeacfb6bd9-kube-api-access-tqmgc\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:39:10.781866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.781842    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b90089-4287-45be-a343-b3ed069e0188-maas-api-xks-ca" pod "40b90089-4287-45be-a343-b3ed069e0188" (UID: "40b90089-4287-45be-a343-b3ed069e0188"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:39:10.781999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.781893    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-xks-ca" pod "d477fb64-5ef6-4e61-977f-4477417704f6" (UID: "d477fb64-5ef6-4e61-977f-4477417704f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:39:10.781999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.781905    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b90089-4287-45be-a343-b3ed069e0188-kube-api-access-htk7z" pod "40b90089-4287-45be-a343-b3ed069e0188" (UID: "40b90089-4287-45be-a343-b3ed069e0188"). InnerVolumeSpecName "kube-api-access-htk7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:39:10.782281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.782262    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d477fb64-5ef6-4e61-977f-4477417704f6-kube-api-access-z94rz" pod "d477fb64-5ef6-4e61-977f-4477417704f6" (UID: "d477fb64-5ef6-4e61-977f-4477417704f6"). InnerVolumeSpecName "kube-api-access-z94rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:39:10.882108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.882066    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b90089-4287-45be-a343-b3ed069e0188-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:39:10.882108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.882102    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z94rz\" (UniqueName: \"kubernetes.io/projected/d477fb64-5ef6-4e61-977f-4477417704f6-kube-api-access-z94rz\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:39:10.882108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.882114    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htk7z\" (UniqueName: \"kubernetes.io/projected/40b90089-4287-45be-a343-b3ed069e0188-kube-api-access-htk7z\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:39:10.882419 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:10.882126    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d477fb64-5ef6-4e61-977f-4477417704f6-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:39:11.379723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:11.379695    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"
Nov 03 07:39:11.379910 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:11.379727    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn" event={"ID":"d477fb64-5ef6-4e61-977f-4477417704f6","Type":"ContainerDied","Data":"78c86eee3a53174b447a99b0549c3fe39ba9927b7bda787abe5d55a24ebfa73b"}
Nov 03 07:39:11.379910 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:11.379779    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78c86eee3a53174b447a99b0549c3fe39ba9927b7bda787abe5d55a24ebfa73b"
Nov 03 07:39:11.381198 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:11.381170    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j" event={"ID":"40b90089-4287-45be-a343-b3ed069e0188","Type":"ContainerDied","Data":"760801dc53a7f8b3d139b270d523c6195871a8ad627f2522c43bb38fdf93a326"}
Nov 03 07:39:11.381198 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:11.381193    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"
Nov 03 07:39:11.381364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:11.381213    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="760801dc53a7f8b3d139b270d523c6195871a8ad627f2522c43bb38fdf93a326"
Nov 03 07:39:11.382437 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:11.382417    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5" event={"ID":"ad571469-adaa-4e64-b88a-6adeacfb6bd9","Type":"ContainerDied","Data":"6baedbd82bf5e5caea45f18da260cb32facfc4e1d6dbf464380c4fde3d55c2ee"}
Nov 03 07:39:11.382534 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:11.382447    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6baedbd82bf5e5caea45f18da260cb32facfc4e1d6dbf464380c4fde3d55c2ee"
Nov 03 07:39:11.382534 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:11.382453    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"
Nov 03 07:39:14.185621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:14.185589    4470 scope.go:122] "RemoveContainer" containerID="fa29d8323ac4a065ebe290d463c979c1651abf26c1de109da5f765c3f85fdeb1"
Nov 03 07:39:18.573674 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:18.573633    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"]
Nov 03 07:39:23.574413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:23.574350    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6" podUID="8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7" containerName="payload-pre-processing" containerID="cri-o://92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767" gracePeriod=25
Nov 03 07:39:23.848488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:23.848463    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:39:23.974415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:23.974382    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-plugins-config-volume\") pod \"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7\" (UID: \"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7\") "
Nov 03 07:39:23.974598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:23.974578    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-kube-api-access-wr5zj\" (UniqueName: \"kubernetes.io/projected/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-kube-api-access-wr5zj\") pod \"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7\" (UID: \"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7\") "
Nov 03 07:39:23.974809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:23.974784    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-plugins-config-volume" pod "8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7" (UID: "8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:39:23.975281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:23.975258    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:39:23.976890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:23.976867    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-kube-api-access-wr5zj" pod "8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7" (UID: "8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7"). InnerVolumeSpecName "kube-api-access-wr5zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:39:24.076841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:24.076806    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5zj\" (UniqueName: \"kubernetes.io/projected/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7-kube-api-access-wr5zj\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:39:24.471386 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:24.471346    4470 generic.go:356] "Generic (PLEG): container finished" podID="8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7" containerID="92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767" exitCode=0
Nov 03 07:39:24.471564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:24.471396    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"
Nov 03 07:39:24.471564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:24.471429    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6" event={"ID":"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7","Type":"ContainerDied","Data":"92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767"}
Nov 03 07:39:24.471564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:24.471487    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6" event={"ID":"8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7","Type":"ContainerDied","Data":"4b3af277fbe3a8e8672b99841372b3180b93c1a9e142113ff75f2c2e38929773"}
Nov 03 07:39:24.471564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:24.471516    4470 scope.go:122] "RemoveContainer" containerID="92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767"
Nov 03 07:39:24.482271 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:24.482246    4470 scope.go:122] "RemoveContainer" containerID="92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767"
Nov 03 07:39:24.482553 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:39:24.482530    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767\": container with ID starting with 92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767 not found: ID does not exist" containerID="92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767"
Nov 03 07:39:24.482634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:24.482564    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767"} err="failed to get container status \"92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767\": rpc error: code = NotFound desc = could not find container \"92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767\": container with ID starting with 92cf6503fb4962037028b6a16994339ce446dc98512c92dca1925f4929afe767 not found: ID does not exist"
Nov 03 07:39:24.644983 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:24.644950    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"]
Nov 03 07:39:24.695296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:24.695263    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6"]
Nov 03 07:39:25.806754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:25.806723    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7" path="/var/lib/kubelet/pods/8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7/volumes"
Nov 03 07:39:30.372822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.372784    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"]
Nov 03 07:39:30.373360 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373345    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ad571469-adaa-4e64-b88a-6adeacfb6bd9" containerName="revoke-keys"
Nov 03 07:39:30.373425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373360    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad571469-adaa-4e64-b88a-6adeacfb6bd9" containerName="revoke-keys"
Nov 03 07:39:30.373425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373379    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d477fb64-5ef6-4e61-977f-4477417704f6" containerName="revoke-keys"
Nov 03 07:39:30.373425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373385    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d477fb64-5ef6-4e61-977f-4477417704f6" containerName="revoke-keys"
Nov 03 07:39:30.373425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373396    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7" containerName="payload-pre-processing"
Nov 03 07:39:30.373425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373401    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7" containerName="payload-pre-processing"
Nov 03 07:39:30.373425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373410    4470 cpu_manager.go:414] "containerMap: removing container" podUID="40b90089-4287-45be-a343-b3ed069e0188" containerName="revoke-keys"
Nov 03 07:39:30.373425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373415    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b90089-4287-45be-a343-b3ed069e0188" containerName="revoke-keys"
Nov 03 07:39:30.373688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373541    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad571469-adaa-4e64-b88a-6adeacfb6bd9" containerName="revoke-keys"
Nov 03 07:39:30.373688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373552    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d477fb64-5ef6-4e61-977f-4477417704f6" containerName="revoke-keys"
Nov 03 07:39:30.373688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373558    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a71dad7-b3b4-4b7b-b69b-c81585ef7ab7" containerName="payload-pre-processing"
Nov 03 07:39:30.373688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.373566    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b90089-4287-45be-a343-b3ed069e0188" containerName="revoke-keys"
Nov 03 07:39:30.378490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.378466    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.442667 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.438499    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-18b89b-openshift-default-dockercfg-nzt4z\""
Nov 03 07:39:30.464159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.462118    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-credential-socket\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.464159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.462188    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-socket\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.464159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.462222    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-certs\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.464159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.462252    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-envoy\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.464159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.462292    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-data\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.464159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.462323    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8v8f\" (UniqueName: \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-kube-api-access-n8v8f\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.464159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.462371    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d336591b-65cf-4a16-a050-569193c8f481-istio-podinfo\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.464159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.462460    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d336591b-65cf-4a16-a050-569193c8f481-istiod-ca-cert\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.464159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.462663    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-istio-token\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.585178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.585115    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"]
Nov 03 07:39:30.586256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.586225    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-data\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.586435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.586417    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8v8f\" (UniqueName: \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-kube-api-access-n8v8f\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.586566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.586554    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d336591b-65cf-4a16-a050-569193c8f481-istio-podinfo\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.586718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.586706    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d336591b-65cf-4a16-a050-569193c8f481-istiod-ca-cert\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.586929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.586914    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-istio-token\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.587068 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.587056    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-credential-socket\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.587181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.587168    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-socket\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.587317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.587296    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-certs\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.587498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.587480    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-envoy\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.590049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.588793    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-socket\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.590049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.589007    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-certs\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.590049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.589526    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-data\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.590940 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.590700    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-envoy\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.590940 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.590899    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-credential-socket\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.593821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.593803    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d336591b-65cf-4a16-a050-569193c8f481-istiod-ca-cert\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.595963 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.595911    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d336591b-65cf-4a16-a050-569193c8f481-istio-podinfo\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.653860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.653789    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-istio-token\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.654017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.653993    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8v8f\" (UniqueName: \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-kube-api-access-n8v8f\") pod \"e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") " pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:30.748998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:30.748966    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:31.182266 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:31.182238    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"]
Nov 03 07:39:31.182565 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:39:31.182546    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd336591b_65cf_4a16_a050_569193c8f481.slice/crio-def44a8c050e53da15debdb6c93bfc67cb1249684450f5b726c1fab00fc0f738 WatchSource:0}: Error finding container def44a8c050e53da15debdb6c93bfc67cb1249684450f5b726c1fab00fc0f738: Status 404 returned error can't find the container with id def44a8c050e53da15debdb6c93bfc67cb1249684450f5b726c1fab00fc0f738
Nov 03 07:39:31.185070 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:31.185036    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:39:31.185281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:31.185099    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:39:31.185281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:31.185128    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:39:31.609766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:31.609729    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" event={"ID":"d336591b-65cf-4a16-a050-569193c8f481","Type":"ContainerStarted","Data":"cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d"}
Nov 03 07:39:31.610268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:31.609797    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" event={"ID":"d336591b-65cf-4a16-a050-569193c8f481","Type":"ContainerStarted","Data":"def44a8c050e53da15debdb6c93bfc67cb1249684450f5b726c1fab00fc0f738"}
Nov 03 07:39:31.752157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:31.745246    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" podStartSLOduration=1.745227881 podStartE2EDuration="1.745227881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:31.74341148 +0000 UTC m=+2424.075780478" watchObservedRunningTime="2026-11-03 07:39:31.745227881 +0000 UTC m=+2424.077596876"
Nov 03 07:39:31.752462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:31.752436    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:32.756116 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:32.756084    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:33.623008 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:33.622975    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:33.624081 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:33.624062    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:39:34.760199 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:39:34.760164    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:39:37.415535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.415505    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"]
Nov 03 07:39:37.419688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.419663    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.473990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.473953    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-socket\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.474209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.474017    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-token\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.474209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.474066    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm8xf\" (UniqueName: \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-kube-api-access-tm8xf\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.474209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.474096    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-credential-socket\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.474365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.474206    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-envoy\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.474365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.474259    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-certs\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.474365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.474359    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8605b360-6f53-41b3-a1ab-295c6bbdd046-istiod-ca-cert\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.474487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.474406    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-data\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.474487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.474469    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-podinfo\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.475220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.475198    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-44c08f-openshift-default-dockercfg-fzhff\""
Nov 03 07:39:37.488569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.488545    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"]
Nov 03 07:39:37.575684 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.575649    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-podinfo\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.575684 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.575688    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-socket\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.575942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.575734    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-token\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.575942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.575749    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm8xf\" (UniqueName: \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-kube-api-access-tm8xf\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.575942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.575779    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-credential-socket\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.575942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.575826    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-envoy\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.575942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.575858    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-certs\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.576340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.575974    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8605b360-6f53-41b3-a1ab-295c6bbdd046-istiod-ca-cert\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.576340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.576048    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-data\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.576340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.576266    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-socket\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.576454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.576339    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-certs\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.576454 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.576443    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-data\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.576530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.576499    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-credential-socket\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.576669 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.576647    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8605b360-6f53-41b3-a1ab-295c6bbdd046-istiod-ca-cert\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.578800 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.578769    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-envoy\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.578911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.578869    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-podinfo\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.640942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.640889    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm8xf\" (UniqueName: \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-kube-api-access-tm8xf\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.648481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.647690    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-token\") pod \"e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") " pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:37.736540 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:37.736457    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:38.170165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:38.170123    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"]
Nov 03 07:39:38.171397 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:39:38.171374    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8605b360_6f53_41b3_a1ab_295c6bbdd046.slice/crio-68dc2a0f7f8dbe17ef8fad93ab729324326b7adeaf89467238e51c13e82919b3 WatchSource:0}: Error finding container 68dc2a0f7f8dbe17ef8fad93ab729324326b7adeaf89467238e51c13e82919b3: Status 404 returned error can't find the container with id 68dc2a0f7f8dbe17ef8fad93ab729324326b7adeaf89467238e51c13e82919b3
Nov 03 07:39:38.174180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:38.174126    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:39:38.174262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:38.174212    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:39:38.174262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:38.174244    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:39:38.663903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:38.663864    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" event={"ID":"8605b360-6f53-41b3-a1ab-295c6bbdd046","Type":"ContainerStarted","Data":"c327cf187cec1f3259aca10154dc78afda76cd2bbd1110ee0e92519a1b2c21da"}
Nov 03 07:39:38.664461 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:38.663928    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" event={"ID":"8605b360-6f53-41b3-a1ab-295c6bbdd046","Type":"ContainerStarted","Data":"68dc2a0f7f8dbe17ef8fad93ab729324326b7adeaf89467238e51c13e82919b3"}
Nov 03 07:39:38.737165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:38.737099    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:38.840691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:38.840636    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" podStartSLOduration=1.840620905 podStartE2EDuration="1.840620905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:38.838568485 +0000 UTC m=+2431.170937481" watchObservedRunningTime="2026-11-03 07:39:38.840620905 +0000 UTC m=+2431.172989974"
Nov 03 07:39:39.840495 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:39.840460    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:40.677803 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:40.677769    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:40.678933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:40.678915    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:39:48.215341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.215309    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"]
Nov 03 07:39:48.220744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.220723    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:39:48.277724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.277689    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"]
Nov 03 07:39:48.282430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.282395    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhj92\" (UniqueName: \"kubernetes.io/projected/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-kube-api-access-mhj92\") pod \"payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn\" (UID: \"d5b6d9c1-75ff-455c-9b37-8df77fad98cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:39:48.282590 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.282469    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn\" (UID: \"d5b6d9c1-75ff-455c-9b37-8df77fad98cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:39:48.324920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.324870    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-44c08f\""
Nov 03 07:39:48.325089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.324983    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-44c08f-dockercfg-2k5fn\""
Nov 03 07:39:48.386977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.386932    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhj92\" (UniqueName: \"kubernetes.io/projected/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-kube-api-access-mhj92\") pod \"payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn\" (UID: \"d5b6d9c1-75ff-455c-9b37-8df77fad98cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:39:48.387253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.387046    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn\" (UID: \"d5b6d9c1-75ff-455c-9b37-8df77fad98cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:39:48.400206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.388234    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn\" (UID: \"d5b6d9c1-75ff-455c-9b37-8df77fad98cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:39:48.416296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.416212    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"]
Nov 03 07:39:48.456900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.456870    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"]
Nov 03 07:39:48.457359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.457217    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:39:48.491295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.485690    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhj92\" (UniqueName: \"kubernetes.io/projected/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-kube-api-access-mhj92\") pod \"payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn\" (UID: \"d5b6d9c1-75ff-455c-9b37-8df77fad98cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:39:48.492809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.492660    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv2mw\" (UniqueName: \"kubernetes.io/projected/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-kube-api-access-gv2mw\") pod \"payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h\" (UID: \"c1eb7828-42fd-4c5a-80c8-a86cce97fd10\") " pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:39:48.492964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.492931    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-plugins-config-volume\") pod \"payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h\" (UID: \"c1eb7828-42fd-4c5a-80c8-a86cce97fd10\") " pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:39:48.536766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.536727    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:39:48.594595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.594557    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv2mw\" (UniqueName: \"kubernetes.io/projected/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-kube-api-access-gv2mw\") pod \"payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h\" (UID: \"c1eb7828-42fd-4c5a-80c8-a86cce97fd10\") " pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:39:48.594958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.594933    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-plugins-config-volume\") pod \"payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h\" (UID: \"c1eb7828-42fd-4c5a-80c8-a86cce97fd10\") " pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:39:48.595611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.595593    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-plugins-config-volume\") pod \"payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h\" (UID: \"c1eb7828-42fd-4c5a-80c8-a86cce97fd10\") " pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:39:48.658818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.658785    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv2mw\" (UniqueName: \"kubernetes.io/projected/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-kube-api-access-gv2mw\") pod \"payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h\" (UID: \"c1eb7828-42fd-4c5a-80c8-a86cce97fd10\") " pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:39:48.817399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.817367    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:39:48.927631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:48.927600    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"]
Nov 03 07:39:48.929175 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:39:48.929126    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b6d9c1_75ff_455c_9b37_8df77fad98cb.slice/crio-349b7541c82235eabda972e23ea7583d9faee6f38df4f65ea741ed2cc8223bee WatchSource:0}: Error finding container 349b7541c82235eabda972e23ea7583d9faee6f38df4f65ea741ed2cc8223bee: Status 404 returned error can't find the container with id 349b7541c82235eabda972e23ea7583d9faee6f38df4f65ea741ed2cc8223bee
Nov 03 07:39:49.198020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:49.197985    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"]
Nov 03 07:39:49.198322 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:39:49.198290    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1eb7828_42fd_4c5a_80c8_a86cce97fd10.slice/crio-a279b142ddba0328a648be709f4ccefca235a24a3482a31f26555431d63d084b WatchSource:0}: Error finding container a279b142ddba0328a648be709f4ccefca235a24a3482a31f26555431d63d084b: Status 404 returned error can't find the container with id a279b142ddba0328a648be709f4ccefca235a24a3482a31f26555431d63d084b
Nov 03 07:39:49.740787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:49.740749    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h" event={"ID":"c1eb7828-42fd-4c5a-80c8-a86cce97fd10","Type":"ContainerStarted","Data":"d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741"}
Nov 03 07:39:49.741287 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:49.740809    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h" event={"ID":"c1eb7828-42fd-4c5a-80c8-a86cce97fd10","Type":"ContainerStarted","Data":"a279b142ddba0328a648be709f4ccefca235a24a3482a31f26555431d63d084b"}
Nov 03 07:39:49.741287 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:49.740841    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:39:49.742306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:49.742281    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn" event={"ID":"d5b6d9c1-75ff-455c-9b37-8df77fad98cb","Type":"ContainerStarted","Data":"888181339daa6cb190b2a595af176575815dd99d0180e0ea8922a693697f983b"}
Nov 03 07:39:49.742435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:49.742333    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn" event={"ID":"d5b6d9c1-75ff-455c-9b37-8df77fad98cb","Type":"ContainerStarted","Data":"349b7541c82235eabda972e23ea7583d9faee6f38df4f65ea741ed2cc8223bee"}
Nov 03 07:39:49.742435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:49.742362    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:39:49.864745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:49.864689    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h" podStartSLOduration=1.864674644 podStartE2EDuration="1.864674644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:49.862781859 +0000 UTC m=+2442.195150854" watchObservedRunningTime="2026-11-03 07:39:49.864674644 +0000 UTC m=+2442.197043638"
Nov 03 07:39:55.606955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.606899    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn" podStartSLOduration=7.606880971 podStartE2EDuration="7.606880971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:49.999191781 +0000 UTC m=+2442.331560776" watchObservedRunningTime="2026-11-03 07:39:55.606880971 +0000 UTC m=+2447.939249966"
Nov 03 07:39:55.608416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.608397    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"]
Nov 03 07:39:55.612727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.612689    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:39:55.671439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.671405    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"]
Nov 03 07:39:55.721159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.720472    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-eac348\""
Nov 03 07:39:55.721159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.720618    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-eac348-dockercfg-tbblg\""
Nov 03 07:39:55.759157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.757480    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxnbt\" (UniqueName: \"kubernetes.io/projected/841d918d-896d-4796-a0f6-546daac3f6bd-kube-api-access-rxnbt\") pod \"payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh\" (UID: \"841d918d-896d-4796-a0f6-546daac3f6bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:39:55.759157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.757545    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841d918d-896d-4796-a0f6-546daac3f6bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh\" (UID: \"841d918d-896d-4796-a0f6-546daac3f6bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:39:55.827159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.826682    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"]
Nov 03 07:39:55.839157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.833017    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:39:55.842241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.842203    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"]
Nov 03 07:39:55.859243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.859178    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxnbt\" (UniqueName: \"kubernetes.io/projected/841d918d-896d-4796-a0f6-546daac3f6bd-kube-api-access-rxnbt\") pod \"payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh\" (UID: \"841d918d-896d-4796-a0f6-546daac3f6bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:39:55.859243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.859222    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841d918d-896d-4796-a0f6-546daac3f6bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh\" (UID: \"841d918d-896d-4796-a0f6-546daac3f6bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:39:55.859979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.859953    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841d918d-896d-4796-a0f6-546daac3f6bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh\" (UID: \"841d918d-896d-4796-a0f6-546daac3f6bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:39:55.923661 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.923625    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxnbt\" (UniqueName: \"kubernetes.io/projected/841d918d-896d-4796-a0f6-546daac3f6bd-kube-api-access-rxnbt\") pod \"payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh\" (UID: \"841d918d-896d-4796-a0f6-546daac3f6bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:39:55.926536 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.926500    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:39:55.961084 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.961038    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqx9m\" (UniqueName: \"kubernetes.io/projected/7832737d-2343-4dfa-ba59-531727ea3ec1-kube-api-access-zqx9m\") pod \"payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n\" (UID: \"7832737d-2343-4dfa-ba59-531727ea3ec1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:39:55.961346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:55.961305    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7832737d-2343-4dfa-ba59-531727ea3ec1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n\" (UID: \"7832737d-2343-4dfa-ba59-531727ea3ec1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:39:56.063239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.063189    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqx9m\" (UniqueName: \"kubernetes.io/projected/7832737d-2343-4dfa-ba59-531727ea3ec1-kube-api-access-zqx9m\") pod \"payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n\" (UID: \"7832737d-2343-4dfa-ba59-531727ea3ec1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:39:56.063622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.063325    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7832737d-2343-4dfa-ba59-531727ea3ec1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n\" (UID: \"7832737d-2343-4dfa-ba59-531727ea3ec1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:39:56.064044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.064019    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7832737d-2343-4dfa-ba59-531727ea3ec1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n\" (UID: \"7832737d-2343-4dfa-ba59-531727ea3ec1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:39:56.127718 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.127637    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqx9m\" (UniqueName: \"kubernetes.io/projected/7832737d-2343-4dfa-ba59-531727ea3ec1-kube-api-access-zqx9m\") pod \"payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n\" (UID: \"7832737d-2343-4dfa-ba59-531727ea3ec1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:39:56.153472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.153424    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:39:56.310517 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.310482    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"]
Nov 03 07:39:56.311296 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:39:56.311270    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841d918d_896d_4796_a0f6_546daac3f6bd.slice/crio-a94d4a56ac7dd8ae59fe3a56667e4b8405f713dae1813bd317c127f9e51f1cf4 WatchSource:0}: Error finding container a94d4a56ac7dd8ae59fe3a56667e4b8405f713dae1813bd317c127f9e51f1cf4: Status 404 returned error can't find the container with id a94d4a56ac7dd8ae59fe3a56667e4b8405f713dae1813bd317c127f9e51f1cf4
Nov 03 07:39:56.541981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.541957    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"]
Nov 03 07:39:56.543735 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:39:56.543709    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7832737d_2343_4dfa_ba59_531727ea3ec1.slice/crio-0b9b5a4e3c18d4a19cc6b20716021dd4f5e3e37e7c820936d47092c704e85c34 WatchSource:0}: Error finding container 0b9b5a4e3c18d4a19cc6b20716021dd4f5e3e37e7c820936d47092c704e85c34: Status 404 returned error can't find the container with id 0b9b5a4e3c18d4a19cc6b20716021dd4f5e3e37e7c820936d47092c704e85c34
Nov 03 07:39:56.823879 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.823839    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh" event={"ID":"841d918d-896d-4796-a0f6-546daac3f6bd","Type":"ContainerStarted","Data":"4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317"}
Nov 03 07:39:56.824405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.823916    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:39:56.824405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.823937    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh" event={"ID":"841d918d-896d-4796-a0f6-546daac3f6bd","Type":"ContainerStarted","Data":"a94d4a56ac7dd8ae59fe3a56667e4b8405f713dae1813bd317c127f9e51f1cf4"}
Nov 03 07:39:56.825625 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.825592    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n" event={"ID":"7832737d-2343-4dfa-ba59-531727ea3ec1","Type":"ContainerStarted","Data":"f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016"}
Nov 03 07:39:56.825745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.825639    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n" event={"ID":"7832737d-2343-4dfa-ba59-531727ea3ec1","Type":"ContainerStarted","Data":"0b9b5a4e3c18d4a19cc6b20716021dd4f5e3e37e7c820936d47092c704e85c34"}
Nov 03 07:39:56.825745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.825659    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:39:56.944885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:56.944831    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh" podStartSLOduration=1.944817279 podStartE2EDuration="1.944817279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:56.944097417 +0000 UTC m=+2449.276466412" watchObservedRunningTime="2026-11-03 07:39:56.944817279 +0000 UTC m=+2449.277186273"
Nov 03 07:39:57.072998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:39:57.072930    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n" podStartSLOduration=2.072905912 podStartE2EDuration="2.072905912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:57.066778784 +0000 UTC m=+2449.399147779" watchObservedRunningTime="2026-11-03 07:39:57.072905912 +0000 UTC m=+2449.405274930"
Nov 03 07:40:00.753358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:00.753327    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:40:00.753852 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:00.753410    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:40:04.760002 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:40:04.759962    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:40:05.192480 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:05.192440    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"]
Nov 03 07:40:05.192756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:05.192709    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj" podUID="68028204-1304-4646-8478-6a418f7c0144" containerName="istio-proxy" containerID="cri-o://f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644" gracePeriod=30
Nov 03 07:40:05.196965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:05.196941    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"]
Nov 03 07:40:05.197177 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:05.197155    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy" containerID="cri-o://a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1" gracePeriod=30
Nov 03 07:40:05.233943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:05.233901    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"]
Nov 03 07:40:05.234180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:05.234151    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9" podUID="ee20c586-082c-4333-be74-5e8d49f84711" containerName="istio-proxy" containerID="cri-o://5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e" gracePeriod=30
Nov 03 07:40:07.835336 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:07.835301    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:40:07.835834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:07.835677    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:40:08.172149 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:08.172048    4470 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:40:08.172311 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:08.172115    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:40:10.525436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.525406    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:40:10.548885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.548863    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:40:10.576146 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576101    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68028204-1304-4646-8478-6a418f7c0144-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68028204-1304-4646-8478-6a418f7c0144-istiod-ca-cert\") pod \"68028204-1304-4646-8478-6a418f7c0144\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") "
Nov 03 07:40:10.576338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576159    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-credential-socket\") pod \"68028204-1304-4646-8478-6a418f7c0144\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") "
Nov 03 07:40:10.576338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576284    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-envoy\") pod \"68028204-1304-4646-8478-6a418f7c0144\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") "
Nov 03 07:40:10.576338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576311    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-data\") pod \"68028204-1304-4646-8478-6a418f7c0144\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") "
Nov 03 07:40:10.576585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576457    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68028204-1304-4646-8478-6a418f7c0144-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68028204-1304-4646-8478-6a418f7c0144-istio-podinfo\") pod \"68028204-1304-4646-8478-6a418f7c0144\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") "
Nov 03 07:40:10.576585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576454    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-credential-socket" pod "68028204-1304-4646-8478-6a418f7c0144" (UID: "68028204-1304-4646-8478-6a418f7c0144"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.576585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576486    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-istio-token\" (UniqueName: \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-istio-token\") pod \"68028204-1304-4646-8478-6a418f7c0144\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") "
Nov 03 07:40:10.576585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576522    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-socket\") pod \"68028204-1304-4646-8478-6a418f7c0144\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") "
Nov 03 07:40:10.576585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576526    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68028204-1304-4646-8478-6a418f7c0144-istiod-ca-cert" pod "68028204-1304-4646-8478-6a418f7c0144" (UID: "68028204-1304-4646-8478-6a418f7c0144"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:40:10.576585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576549    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-kube-api-access-5m88c\" (UniqueName: \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-kube-api-access-5m88c\") pod \"68028204-1304-4646-8478-6a418f7c0144\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") "
Nov 03 07:40:10.576863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576646    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-data" pod "68028204-1304-4646-8478-6a418f7c0144" (UID: "68028204-1304-4646-8478-6a418f7c0144"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.576863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576679    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-certs\") pod \"68028204-1304-4646-8478-6a418f7c0144\" (UID: \"68028204-1304-4646-8478-6a418f7c0144\") "
Nov 03 07:40:10.576863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.576836    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-socket" pod "68028204-1304-4646-8478-6a418f7c0144" (UID: "68028204-1304-4646-8478-6a418f7c0144"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.577208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.577023    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-certs" pod "68028204-1304-4646-8478-6a418f7c0144" (UID: "68028204-1304-4646-8478-6a418f7c0144"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.577515 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.577467    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.577515 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.577499    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.577515 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.577515    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68028204-1304-4646-8478-6a418f7c0144-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.577726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.577533    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.577726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.577548    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.578724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.578697    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68028204-1304-4646-8478-6a418f7c0144-istio-podinfo" pod "68028204-1304-4646-8478-6a418f7c0144" (UID: "68028204-1304-4646-8478-6a418f7c0144"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:40:10.578821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.578776    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-envoy" pod "68028204-1304-4646-8478-6a418f7c0144" (UID: "68028204-1304-4646-8478-6a418f7c0144"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.579120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.579093    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-kube-api-access-5m88c" pod "68028204-1304-4646-8478-6a418f7c0144" (UID: "68028204-1304-4646-8478-6a418f7c0144"). InnerVolumeSpecName "kube-api-access-5m88c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:40:10.579291 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.579268    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-istio-token" pod "68028204-1304-4646-8478-6a418f7c0144" (UID: "68028204-1304-4646-8478-6a418f7c0144"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:40:10.587344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.587323    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:40:10.678680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.678633    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee20c586-082c-4333-be74-5e8d49f84711-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee20c586-082c-4333-be74-5e8d49f84711-istio-podinfo\") pod \"ee20c586-082c-4333-be74-5e8d49f84711\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") "
Nov 03 07:40:10.678893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.678715    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-envoy\") pod \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") "
Nov 03 07:40:10.678893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.678774    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-credential-socket\") pod \"ee20c586-082c-4333-be74-5e8d49f84711\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") "
Nov 03 07:40:10.678893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.678818    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-token\" (UniqueName: \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-token\") pod \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") "
Nov 03 07:40:10.678893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.678879    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istiod-ca-cert\") pod \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") "
Nov 03 07:40:10.679165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.678921    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-istio-token\" (UniqueName: \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-istio-token\") pod \"ee20c586-082c-4333-be74-5e8d49f84711\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") "
Nov 03 07:40:10.679165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.678949    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-certs\") pod \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") "
Nov 03 07:40:10.679165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679035    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-podinfo\") pod \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") "
Nov 03 07:40:10.679165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679063    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-certs\") pod \"ee20c586-082c-4333-be74-5e8d49f84711\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") "
Nov 03 07:40:10.679165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679105    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-data\") pod \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") "
Nov 03 07:40:10.679165 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679105    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-credential-socket" pod "ee20c586-082c-4333-be74-5e8d49f84711" (UID: "ee20c586-082c-4333-be74-5e8d49f84711"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.679509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679169    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-kube-api-access-46mqb\" (UniqueName: \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-kube-api-access-46mqb\") pod \"ee20c586-082c-4333-be74-5e8d49f84711\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") "
Nov 03 07:40:10.679509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679200    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee20c586-082c-4333-be74-5e8d49f84711-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee20c586-082c-4333-be74-5e8d49f84711-istiod-ca-cert\") pod \"ee20c586-082c-4333-be74-5e8d49f84711\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") "
Nov 03 07:40:10.679509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679255    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-socket\") pod \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") "
Nov 03 07:40:10.679509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679330    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-envoy\") pod \"ee20c586-082c-4333-be74-5e8d49f84711\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") "
Nov 03 07:40:10.679509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679346    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-certs" pod "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" (UID: "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.679509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679379    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-socket\") pod \"ee20c586-082c-4333-be74-5e8d49f84711\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") "
Nov 03 07:40:10.679509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679440    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-credential-socket\") pod \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") "
Nov 03 07:40:10.679509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679488    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-data\") pod \"ee20c586-082c-4333-be74-5e8d49f84711\" (UID: \"ee20c586-082c-4333-be74-5e8d49f84711\") "
Nov 03 07:40:10.679969 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.679535    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-kube-api-access-dkhg2\" (UniqueName: \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-kube-api-access-dkhg2\") pod \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\" (UID: \"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba\") "
Nov 03 07:40:10.680187 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.680167    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68028204-1304-4646-8478-6a418f7c0144-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.680284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.680193    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.680284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.680210    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m88c\" (UniqueName: \"kubernetes.io/projected/68028204-1304-4646-8478-6a418f7c0144-kube-api-access-5m88c\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.680284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.680224    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.680284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.680239    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68028204-1304-4646-8478-6a418f7c0144-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.680284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.680252    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.680284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.680266    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istiod-ca-cert" pod "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" (UID: "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:40:10.680708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.680628    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee20c586-082c-4333-be74-5e8d49f84711-istiod-ca-cert" pod "ee20c586-082c-4333-be74-5e8d49f84711" (UID: "ee20c586-082c-4333-be74-5e8d49f84711"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:40:10.682313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.681156    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-data" pod "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" (UID: "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.682313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.681360    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-certs" pod "ee20c586-082c-4333-be74-5e8d49f84711" (UID: "ee20c586-082c-4333-be74-5e8d49f84711"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.682313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.681541    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-socket" pod "ee20c586-082c-4333-be74-5e8d49f84711" (UID: "ee20c586-082c-4333-be74-5e8d49f84711"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.682313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.681612    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee20c586-082c-4333-be74-5e8d49f84711-istio-podinfo" pod "ee20c586-082c-4333-be74-5e8d49f84711" (UID: "ee20c586-082c-4333-be74-5e8d49f84711"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:40:10.682313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.681814    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-socket" pod "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" (UID: "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.682313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.681983    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-credential-socket" pod "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" (UID: "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.682313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.682036    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-envoy" pod "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" (UID: "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.682313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.682269    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-data" pod "ee20c586-082c-4333-be74-5e8d49f84711" (UID: "ee20c586-082c-4333-be74-5e8d49f84711"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.682939 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.682882    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-kube-api-access-dkhg2" pod "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" (UID: "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba"). InnerVolumeSpecName "kube-api-access-dkhg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:40:10.683169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.683103    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-podinfo" pod "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" (UID: "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:40:10.683607 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.683586    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-token" pod "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" (UID: "a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:40:10.683743 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.683723    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-istio-token" pod "ee20c586-082c-4333-be74-5e8d49f84711" (UID: "ee20c586-082c-4333-be74-5e8d49f84711"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:40:10.684100 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.684080    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-envoy" pod "ee20c586-082c-4333-be74-5e8d49f84711" (UID: "ee20c586-082c-4333-be74-5e8d49f84711"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:10.685221 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.685204    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-kube-api-access-46mqb" pod "ee20c586-082c-4333-be74-5e8d49f84711" (UID: "ee20c586-082c-4333-be74-5e8d49f84711"). InnerVolumeSpecName "kube-api-access-46mqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781823    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781861    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781876    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781890    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781902    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781915    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46mqb\" (UniqueName: \"kubernetes.io/projected/ee20c586-082c-4333-be74-5e8d49f84711-kube-api-access-46mqb\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781929    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee20c586-082c-4333-be74-5e8d49f84711-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781942    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781955    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781968    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781980    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781989    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee20c586-082c-4333-be74-5e8d49f84711-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.781997    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkhg2\" (UniqueName: \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-kube-api-access-dkhg2\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.782005    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee20c586-082c-4333-be74-5e8d49f84711-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.782013    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.784023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.782021    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:10.932600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.931259    4470 generic.go:356] "Generic (PLEG): container finished" podID="68028204-1304-4646-8478-6a418f7c0144" containerID="f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644" exitCode=0
Nov 03 07:40:10.932600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.931357    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"
Nov 03 07:40:10.932600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.931434    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj" event={"ID":"68028204-1304-4646-8478-6a418f7c0144","Type":"ContainerDied","Data":"f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644"}
Nov 03 07:40:10.932600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.931489    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj" event={"ID":"68028204-1304-4646-8478-6a418f7c0144","Type":"ContainerDied","Data":"60f11b43a74406faab2c78033fd0ec25742574ffa6842a91fdfd96f67e2a4500"}
Nov 03 07:40:10.932600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.931521    4470 scope.go:122] "RemoveContainer" containerID="f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644"
Nov 03 07:40:10.934318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.934191    4470 generic.go:356] "Generic (PLEG): container finished" podID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerID="a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1" exitCode=0
Nov 03 07:40:10.934318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.934229    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" event={"ID":"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba","Type":"ContainerDied","Data":"a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1"}
Nov 03 07:40:10.934318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.934292    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j" event={"ID":"a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba","Type":"ContainerDied","Data":"2faaee200de29c952a2efb722cccd78c7ca3511d686c0f86e3db1ed8f05573e6"}
Nov 03 07:40:10.934648 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.934633    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"
Nov 03 07:40:10.949461 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.949435    4470 scope.go:122] "RemoveContainer" containerID="f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644"
Nov 03 07:40:10.954476 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:40:10.954454    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644\": container with ID starting with f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644 not found: ID does not exist" containerID="f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644"
Nov 03 07:40:10.954592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.954487    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644"} err="failed to get container status \"f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644\": rpc error: code = NotFound desc = could not find container \"f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644\": container with ID starting with f327b98145a9dc314ae0cee36f5ce1f08bb2903b750c18cb437638513f547644 not found: ID does not exist"
Nov 03 07:40:10.954592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.954511    4470 scope.go:122] "RemoveContainer" containerID="a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1"
Nov 03 07:40:10.955463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.955441    4470 generic.go:356] "Generic (PLEG): container finished" podID="ee20c586-082c-4333-be74-5e8d49f84711" containerID="5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e" exitCode=0
Nov 03 07:40:10.955529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.955506    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"
Nov 03 07:40:10.955584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.955520    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9" event={"ID":"ee20c586-082c-4333-be74-5e8d49f84711","Type":"ContainerDied","Data":"5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e"}
Nov 03 07:40:10.955633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.955579    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9" event={"ID":"ee20c586-082c-4333-be74-5e8d49f84711","Type":"ContainerDied","Data":"4413069d0cab1815b2edf81b5e0f9ad6595a6ace74988d9efbc5af5acd6c8686"}
Nov 03 07:40:10.965008 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.964991    4470 scope.go:122] "RemoveContainer" containerID="4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f"
Nov 03 07:40:10.974553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.974535    4470 scope.go:122] "RemoveContainer" containerID="a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1"
Nov 03 07:40:10.974858 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:40:10.974830    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1\": container with ID starting with a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1 not found: ID does not exist" containerID="a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1"
Nov 03 07:40:10.974958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.974865    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1"} err="failed to get container status \"a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1\": rpc error: code = NotFound desc = could not find container \"a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1\": container with ID starting with a968e770b1157d34c4c2722c02419f8771529db63aa4f2f1ab8addce614f95c1 not found: ID does not exist"
Nov 03 07:40:10.974958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.974883    4470 scope.go:122] "RemoveContainer" containerID="4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f"
Nov 03 07:40:10.975193 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:40:10.975175    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f\": container with ID starting with 4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f not found: ID does not exist" containerID="4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f"
Nov 03 07:40:10.975251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.975198    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f"} err="failed to get container status \"4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f\": rpc error: code = NotFound desc = could not find container \"4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f\": container with ID starting with 4e90bd7d9f3eff8c81987330222e961f5e8e5ebd96b449376ca98e4df78b743f not found: ID does not exist"
Nov 03 07:40:10.975251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.975214    4470 scope.go:122] "RemoveContainer" containerID="5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e"
Nov 03 07:40:10.984307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.984287    4470 scope.go:122] "RemoveContainer" containerID="5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e"
Nov 03 07:40:10.984572 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:40:10.984550    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e\": container with ID starting with 5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e not found: ID does not exist" containerID="5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e"
Nov 03 07:40:10.984626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:10.984579    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e"} err="failed to get container status \"5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e\": rpc error: code = NotFound desc = could not find container \"5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e\": container with ID starting with 5c3673a68f2c7c83203835f6ba15a74a1cd47551ebfc84c93a9773c12cd1f68e not found: ID does not exist"
Nov 03 07:40:11.103589 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:11.103560    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"]
Nov 03 07:40:11.109366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:11.109337    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj"]
Nov 03 07:40:11.225787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:11.225471    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"]
Nov 03 07:40:11.231265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:11.231238    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j"]
Nov 03 07:40:11.351278 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:11.351240    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"]
Nov 03 07:40:11.357240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:11.357173    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9"]
Nov 03 07:40:11.808825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:11.808739    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68028204-1304-4646-8478-6a418f7c0144" path="/var/lib/kubelet/pods/68028204-1304-4646-8478-6a418f7c0144/volumes"
Nov 03 07:40:11.809242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:11.809228    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" path="/var/lib/kubelet/pods/a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba/volumes"
Nov 03 07:40:11.809680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:11.809667    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee20c586-082c-4333-be74-5e8d49f84711" path="/var/lib/kubelet/pods/ee20c586-082c-4333-be74-5e8d49f84711/volumes"
Nov 03 07:40:13.451909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.451868    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"]
Nov 03 07:40:13.452700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.452674    4470 cpu_manager.go:414] "containerMap: removing container" podUID="68028204-1304-4646-8478-6a418f7c0144" containerName="istio-proxy"
Nov 03 07:40:13.452700 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.452700    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68028204-1304-4646-8478-6a418f7c0144" containerName="istio-proxy"
Nov 03 07:40:13.452884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.452717    4470 cpu_manager.go:414] "containerMap: removing container" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy"
Nov 03 07:40:13.452884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.452723    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy"
Nov 03 07:40:13.452884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.452739    4470 cpu_manager.go:414] "containerMap: removing container" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy"
Nov 03 07:40:13.452884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.452746    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy"
Nov 03 07:40:13.452884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.452773    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ee20c586-082c-4333-be74-5e8d49f84711" containerName="istio-proxy"
Nov 03 07:40:13.452884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.452778    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee20c586-082c-4333-be74-5e8d49f84711" containerName="istio-proxy"
Nov 03 07:40:13.452884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.452877    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee20c586-082c-4333-be74-5e8d49f84711" containerName="istio-proxy"
Nov 03 07:40:13.453230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.452890    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="68028204-1304-4646-8478-6a418f7c0144" containerName="istio-proxy"
Nov 03 07:40:13.453230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.452898    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy"
Nov 03 07:40:13.458259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.458240    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:13.504665 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.504627    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"]
Nov 03 07:40:13.505417 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.505401    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92f7a7c-240f-4bb9-a1e3-2fe9f653c0ba" containerName="istio-proxy"
Nov 03 07:40:13.508885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.508869    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:13.552444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.552400    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:40:13.552444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.552406    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:40:13.552723 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.552471    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:40:13.563732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.563699    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:40:13.568440 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.568408    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"]
Nov 03 07:40:13.572265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.572241    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"]
Nov 03 07:40:13.620069 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.620033    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqzw\" (UniqueName: \"kubernetes.io/projected/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-kube-api-access-lqqzw\") pod \"maas-api-revoke-sub-bd5374f9-hsb9w\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:13.620280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.620166    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8da554f-kpg22\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:13.620280 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.620236    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8da554f-kpg22\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:13.620355 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.620293    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd5374f9-hsb9w\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:13.620406 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.620365    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd5374f9-hsb9w\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:13.620455 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.620429    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5cvp\" (UniqueName: \"kubernetes.io/projected/ef2d9cb9-8218-43ca-a05f-20c711e594f6-kube-api-access-w5cvp\") pod \"maas-api-revoke-sub-b8da554f-kpg22\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:13.721393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.721287    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5cvp\" (UniqueName: \"kubernetes.io/projected/ef2d9cb9-8218-43ca-a05f-20c711e594f6-kube-api-access-w5cvp\") pod \"maas-api-revoke-sub-b8da554f-kpg22\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:13.721393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.721389    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqzw\" (UniqueName: \"kubernetes.io/projected/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-kube-api-access-lqqzw\") pod \"maas-api-revoke-sub-bd5374f9-hsb9w\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:13.721658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.721442    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8da554f-kpg22\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:13.721658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.721507    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8da554f-kpg22\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:13.721658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.721531    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd5374f9-hsb9w\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:13.721658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.721579    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd5374f9-hsb9w\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:13.722391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.722363    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd5374f9-hsb9w\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:13.722508 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.722363    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8da554f-kpg22\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:13.724099 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.724077    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd5374f9-hsb9w\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:13.724217 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.724124    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8da554f-kpg22\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:13.785041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.785004    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqzw\" (UniqueName: \"kubernetes.io/projected/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-kube-api-access-lqqzw\") pod \"maas-api-revoke-sub-bd5374f9-hsb9w\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:13.785041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.785021    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5cvp\" (UniqueName: \"kubernetes.io/projected/ef2d9cb9-8218-43ca-a05f-20c711e594f6-kube-api-access-w5cvp\") pod \"maas-api-revoke-sub-b8da554f-kpg22\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:13.822554 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:13.822524    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:14.077327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:14.077297    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:14.204439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:14.204417    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"]
Nov 03 07:40:14.207045 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:40:14.207007    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc1750b_bfa3_4e1b_a24f_a3ca122f01a7.slice/crio-a919a1ed539e4903de74367329135e954423a2c4dd85f39f78bca907471c2c95 WatchSource:0}: Error finding container a919a1ed539e4903de74367329135e954423a2c4dd85f39f78bca907471c2c95: Status 404 returned error can't find the container with id a919a1ed539e4903de74367329135e954423a2c4dd85f39f78bca907471c2c95
Nov 03 07:40:14.252223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:14.252195    4470 scope.go:122] "RemoveContainer" containerID="5ae0c70df59fca01e9611c590730b96a28d7cba509a5914362dcdb9187cd55a7"
Nov 03 07:40:14.262192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:14.262172    4470 scope.go:122] "RemoveContainer" containerID="0ee636498d12e3bf8e1b8a4625619e73030d85e662028c89203519fb837762bc"
Nov 03 07:40:14.353656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:14.353127    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"]
Nov 03 07:40:14.356774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:14.356747    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3fea3a-2h2km"]
Nov 03 07:40:14.464231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:14.464203    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"]
Nov 03 07:40:14.465398 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:40:14.465375    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2d9cb9_8218_43ca_a05f_20c711e594f6.slice/crio-e72bc92ce02e41fc880508b08618ed68bd52c8f744ec182bfe47d10bcb503531 WatchSource:0}: Error finding container e72bc92ce02e41fc880508b08618ed68bd52c8f744ec182bfe47d10bcb503531: Status 404 returned error can't find the container with id e72bc92ce02e41fc880508b08618ed68bd52c8f744ec182bfe47d10bcb503531
Nov 03 07:40:14.987981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:14.987884    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w" event={"ID":"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7","Type":"ContainerStarted","Data":"f55938319a1482be5093c6b40ffa7fa2a000e03af67fd653a40518b7ede3a77e"}
Nov 03 07:40:14.987981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:14.987948    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w" event={"ID":"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7","Type":"ContainerStarted","Data":"a919a1ed539e4903de74367329135e954423a2c4dd85f39f78bca907471c2c95"}
Nov 03 07:40:14.989697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:14.989670    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22" event={"ID":"ef2d9cb9-8218-43ca-a05f-20c711e594f6","Type":"ContainerStarted","Data":"48ead51f5b5e1ea44984659793fb8d036360647ecbe92cccb41f84ebae648aee"}
Nov 03 07:40:14.989810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:14.989721    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22" event={"ID":"ef2d9cb9-8218-43ca-a05f-20c711e594f6","Type":"ContainerStarted","Data":"e72bc92ce02e41fc880508b08618ed68bd52c8f744ec182bfe47d10bcb503531"}
Nov 03 07:40:15.109185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:15.109115    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w" podStartSLOduration=2.109099966 podStartE2EDuration="2.109099966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:40:15.107482254 +0000 UTC m=+2467.439851258" watchObservedRunningTime="2026-11-03 07:40:15.109099966 +0000 UTC m=+2467.441468997"
Nov 03 07:40:15.230385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:15.230334    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22" podStartSLOduration=2.230320704 podStartE2EDuration="2.230320704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:40:15.226736214 +0000 UTC m=+2467.559105209" watchObservedRunningTime="2026-11-03 07:40:15.230320704 +0000 UTC m=+2467.562689698"
Nov 03 07:40:15.807453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:15.807378    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e47105a-35f2-4d3e-a017-66c0bdcc6ee4" path="/var/lib/kubelet/pods/1e47105a-35f2-4d3e-a017-66c0bdcc6ee4/volumes"
Nov 03 07:40:16.009653 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:16.009614    4470 generic.go:356] "Generic (PLEG): container finished" podID="7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7" containerID="f55938319a1482be5093c6b40ffa7fa2a000e03af67fd653a40518b7ede3a77e" exitCode=0
Nov 03 07:40:16.009847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:16.009693    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w" event={"ID":"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7","Type":"ContainerDied","Data":"f55938319a1482be5093c6b40ffa7fa2a000e03af67fd653a40518b7ede3a77e"}
Nov 03 07:40:16.011423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:16.011395    4470 generic.go:356] "Generic (PLEG): container finished" podID="ef2d9cb9-8218-43ca-a05f-20c711e594f6" containerID="48ead51f5b5e1ea44984659793fb8d036360647ecbe92cccb41f84ebae648aee" exitCode=0
Nov 03 07:40:16.011563 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:16.011435    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22" event={"ID":"ef2d9cb9-8218-43ca-a05f-20c711e594f6","Type":"ContainerDied","Data":"48ead51f5b5e1ea44984659793fb8d036360647ecbe92cccb41f84ebae648aee"}
Nov 03 07:40:17.188415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.188383    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:17.227103 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.227076    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:17.267292 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.267255    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-xks-ca\") pod \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") "
Nov 03 07:40:17.267452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.267405    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-kube-api-access-lqqzw\" (UniqueName: \"kubernetes.io/projected/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-kube-api-access-lqqzw\") pod \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") "
Nov 03 07:40:17.267626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.267607    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-openshift-ca\") pod \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\" (UID: \"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7\") "
Nov 03 07:40:17.267940 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.267884    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-openshift-ca" pod "7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7" (UID: "7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:40:17.268181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.268162    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:17.269638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.269620    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-xks-ca" pod "7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7" (UID: "7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:40:17.269791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.269771    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-kube-api-access-lqqzw" pod "7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7" (UID: "7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7"). InnerVolumeSpecName "kube-api-access-lqqzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:40:17.369370 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.369335    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-xks-ca\") pod \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") "
Nov 03 07:40:17.369522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.369453    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-openshift-ca\") pod \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") "
Nov 03 07:40:17.369522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.369481    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2d9cb9-8218-43ca-a05f-20c711e594f6-kube-api-access-w5cvp\" (UniqueName: \"kubernetes.io/projected/ef2d9cb9-8218-43ca-a05f-20c711e594f6-kube-api-access-w5cvp\") pod \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\" (UID: \"ef2d9cb9-8218-43ca-a05f-20c711e594f6\") "
Nov 03 07:40:17.369797 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.369769    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-openshift-ca" pod "ef2d9cb9-8218-43ca-a05f-20c711e594f6" (UID: "ef2d9cb9-8218-43ca-a05f-20c711e594f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:40:17.370029 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.370010    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:17.370078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.370031    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:17.370078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.370041    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqqzw\" (UniqueName: \"kubernetes.io/projected/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7-kube-api-access-lqqzw\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:17.371546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.371521    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-xks-ca" pod "ef2d9cb9-8218-43ca-a05f-20c711e594f6" (UID: "ef2d9cb9-8218-43ca-a05f-20c711e594f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:40:17.371726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.371709    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2d9cb9-8218-43ca-a05f-20c711e594f6-kube-api-access-w5cvp" pod "ef2d9cb9-8218-43ca-a05f-20c711e594f6" (UID: "ef2d9cb9-8218-43ca-a05f-20c711e594f6"). InnerVolumeSpecName "kube-api-access-w5cvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:40:17.471896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.471861    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef2d9cb9-8218-43ca-a05f-20c711e594f6-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:17.471896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:17.471893    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5cvp\" (UniqueName: \"kubernetes.io/projected/ef2d9cb9-8218-43ca-a05f-20c711e594f6-kube-api-access-w5cvp\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:18.025048 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:18.025019    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"
Nov 03 07:40:18.025269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:18.025021    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w" event={"ID":"7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7","Type":"ContainerDied","Data":"a919a1ed539e4903de74367329135e954423a2c4dd85f39f78bca907471c2c95"}
Nov 03 07:40:18.025269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:18.025178    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a919a1ed539e4903de74367329135e954423a2c4dd85f39f78bca907471c2c95"
Nov 03 07:40:18.026611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:18.026590    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"
Nov 03 07:40:18.026740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:18.026640    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22" event={"ID":"ef2d9cb9-8218-43ca-a05f-20c711e594f6","Type":"ContainerDied","Data":"e72bc92ce02e41fc880508b08618ed68bd52c8f744ec182bfe47d10bcb503531"}
Nov 03 07:40:18.026740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:18.026692    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e72bc92ce02e41fc880508b08618ed68bd52c8f744ec182bfe47d10bcb503531"
Nov 03 07:40:22.565060 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:22.565027    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"]
Nov 03 07:40:22.565501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:22.565272    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" podUID="4ed0fc15-6376-4298-a439-fb8f692b82a7" containerName="istio-proxy" containerID="cri-o://63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4" gracePeriod=30
Nov 03 07:40:27.857257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.857227    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:40:27.900258 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900226    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed0fc15-6376-4298-a439-fb8f692b82a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed0fc15-6376-4298-a439-fb8f692b82a7-istiod-ca-cert\") pod \"4ed0fc15-6376-4298-a439-fb8f692b82a7\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") "
Nov 03 07:40:27.900425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900309    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-credential-socket\") pod \"4ed0fc15-6376-4298-a439-fb8f692b82a7\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") "
Nov 03 07:40:27.900425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900407    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-podinfo\") pod \"4ed0fc15-6376-4298-a439-fb8f692b82a7\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") "
Nov 03 07:40:27.900558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900469    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-kube-api-access-wpsq2\" (UniqueName: \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-kube-api-access-wpsq2\") pod \"4ed0fc15-6376-4298-a439-fb8f692b82a7\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") "
Nov 03 07:40:27.900558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900514    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-certs\") pod \"4ed0fc15-6376-4298-a439-fb8f692b82a7\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") "
Nov 03 07:40:27.900679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900565    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-data\") pod \"4ed0fc15-6376-4298-a439-fb8f692b82a7\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") "
Nov 03 07:40:27.900679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900581    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-credential-socket" pod "4ed0fc15-6376-4298-a439-fb8f692b82a7" (UID: "4ed0fc15-6376-4298-a439-fb8f692b82a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:27.900679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900662    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-token\" (UniqueName: \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-token\") pod \"4ed0fc15-6376-4298-a439-fb8f692b82a7\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") "
Nov 03 07:40:27.900679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900668    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed0fc15-6376-4298-a439-fb8f692b82a7-istiod-ca-cert" pod "4ed0fc15-6376-4298-a439-fb8f692b82a7" (UID: "4ed0fc15-6376-4298-a439-fb8f692b82a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:40:27.900883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900703    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-envoy\") pod \"4ed0fc15-6376-4298-a439-fb8f692b82a7\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") "
Nov 03 07:40:27.900883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900817    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-socket\") pod \"4ed0fc15-6376-4298-a439-fb8f692b82a7\" (UID: \"4ed0fc15-6376-4298-a439-fb8f692b82a7\") "
Nov 03 07:40:27.900883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900830    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-certs" pod "4ed0fc15-6376-4298-a439-fb8f692b82a7" (UID: "4ed0fc15-6376-4298-a439-fb8f692b82a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:27.901112 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.900920    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-data" pod "4ed0fc15-6376-4298-a439-fb8f692b82a7" (UID: "4ed0fc15-6376-4298-a439-fb8f692b82a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:27.901112 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.901080    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-socket" pod "4ed0fc15-6376-4298-a439-fb8f692b82a7" (UID: "4ed0fc15-6376-4298-a439-fb8f692b82a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:27.901690 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.901666    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:27.901854 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.901830    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed0fc15-6376-4298-a439-fb8f692b82a7-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:27.901949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.901859    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:27.901949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.901874    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:27.901949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.901891    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:27.903063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.903037    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-token" pod "4ed0fc15-6376-4298-a439-fb8f692b82a7" (UID: "4ed0fc15-6376-4298-a439-fb8f692b82a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:40:27.903207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.903124    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-kube-api-access-wpsq2" pod "4ed0fc15-6376-4298-a439-fb8f692b82a7" (UID: "4ed0fc15-6376-4298-a439-fb8f692b82a7"). InnerVolumeSpecName "kube-api-access-wpsq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:40:27.903509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.903489    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-podinfo" pod "4ed0fc15-6376-4298-a439-fb8f692b82a7" (UID: "4ed0fc15-6376-4298-a439-fb8f692b82a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:40:27.903603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:27.903547    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-envoy" pod "4ed0fc15-6376-4298-a439-fb8f692b82a7" (UID: "4ed0fc15-6376-4298-a439-fb8f692b82a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:40:28.004288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.004246    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:28.004288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.004286    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpsq2\" (UniqueName: \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-kube-api-access-wpsq2\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:28.004562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.004303    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:28.004562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.004319    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed0fc15-6376-4298-a439-fb8f692b82a7-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:40:28.110907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.108346    4470 generic.go:356] "Generic (PLEG): container finished" podID="4ed0fc15-6376-4298-a439-fb8f692b82a7" containerID="63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4" exitCode=0
Nov 03 07:40:28.110907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.108531    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" event={"ID":"4ed0fc15-6376-4298-a439-fb8f692b82a7","Type":"ContainerDied","Data":"63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4"}
Nov 03 07:40:28.110907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.108583    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf" event={"ID":"4ed0fc15-6376-4298-a439-fb8f692b82a7","Type":"ContainerDied","Data":"01499c59268a33d0db5e16dae68367e6ff430561c168d4cf52b6759c94c304ae"}
Nov 03 07:40:28.110907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.108612    4470 scope.go:122] "RemoveContainer" containerID="63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4"
Nov 03 07:40:28.110907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.108769    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"
Nov 03 07:40:28.188832 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.188802    4470 scope.go:122] "RemoveContainer" containerID="63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4"
Nov 03 07:40:28.189312 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:40:28.189284    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4\": container with ID starting with 63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4 not found: ID does not exist" containerID="63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4"
Nov 03 07:40:28.189415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.189328    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4"} err="failed to get container status \"63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4\": rpc error: code = NotFound desc = could not find container \"63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4\": container with ID starting with 63ee4f2974fec262edf343f365e8401f0d64406241746a7acda17e8677d7a3f4 not found: ID does not exist"
Nov 03 07:40:28.304577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.304542    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"]
Nov 03 07:40:28.310235 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:28.310198    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf"]
Nov 03 07:40:29.826686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:29.826644    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed0fc15-6376-4298-a439-fb8f692b82a7" path="/var/lib/kubelet/pods/4ed0fc15-6376-4298-a439-fb8f692b82a7/volumes"
Nov 03 07:40:34.760602 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:40:34.760562    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:40:55.080230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.080182    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"]
Nov 03 07:40:55.082735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.081184    4470 cpu_manager.go:414] "containerMap: removing container" podUID="4ed0fc15-6376-4298-a439-fb8f692b82a7" containerName="istio-proxy"
Nov 03 07:40:55.082735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.081200    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed0fc15-6376-4298-a439-fb8f692b82a7" containerName="istio-proxy"
Nov 03 07:40:55.082735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.081237    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ef2d9cb9-8218-43ca-a05f-20c711e594f6" containerName="revoke-keys"
Nov 03 07:40:55.082735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.081245    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2d9cb9-8218-43ca-a05f-20c711e594f6" containerName="revoke-keys"
Nov 03 07:40:55.082735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.081266    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7" containerName="revoke-keys"
Nov 03 07:40:55.082735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.081271    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7" containerName="revoke-keys"
Nov 03 07:40:55.082735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.081382    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed0fc15-6376-4298-a439-fb8f692b82a7" containerName="istio-proxy"
Nov 03 07:40:55.082735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.081390    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2d9cb9-8218-43ca-a05f-20c711e594f6" containerName="revoke-keys"
Nov 03 07:40:55.082735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.081400    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7" containerName="revoke-keys"
Nov 03 07:40:55.084883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.084866    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.184204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.184161    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szcml\" (UniqueName: \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-kube-api-access-szcml\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.184398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.184335    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.184398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.184383    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.184495 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.184424    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.184495 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.184448    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8542d8-8120-4a6c-9457-33812c67e0ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.184581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.184551    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.184759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.184740    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.184809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.184795    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.184893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.184874    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.202547 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.202516    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"]
Nov 03 07:40:55.286058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286021    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.286324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286086    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.286324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286203    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.286324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286282    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szcml\" (UniqueName: \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-kube-api-access-szcml\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.286509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286407    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.286509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286433    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.286509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286477    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.286509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286486    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.286509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286501    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.286826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286504    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8542d8-8120-4a6c-9457-33812c67e0ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.286826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286559    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.286826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286565    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.287022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.286975    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.287189 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.287170    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8542d8-8120-4a6c-9457-33812c67e0ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.288728 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.288699    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.289079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.289057    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.389214 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.379303    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szcml\" (UniqueName: \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-kube-api-access-szcml\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.389458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.385061    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gbst7\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.398781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.398450    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:55.881216 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:40:55.881121    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8542d8_8120_4a6c_9457_33812c67e0ae.slice/crio-28a84ae2975448b4c060bc39db415774c2073b817c2dfc94cc50c1ac7ad5d237 WatchSource:0}: Error finding container 28a84ae2975448b4c060bc39db415774c2073b817c2dfc94cc50c1ac7ad5d237: Status 404 returned error can't find the container with id 28a84ae2975448b4c060bc39db415774c2073b817c2dfc94cc50c1ac7ad5d237
Nov 03 07:40:55.884491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.884452    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:40:55.884618 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.884511    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"]
Nov 03 07:40:55.884618 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.884535    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:40:55.884618 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:55.884586    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:40:56.421989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:56.421939    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" event={"ID":"eb8542d8-8120-4a6c-9457-33812c67e0ae","Type":"ContainerStarted","Data":"8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51"}
Nov 03 07:40:56.421989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:56.421996    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" event={"ID":"eb8542d8-8120-4a6c-9457-33812c67e0ae","Type":"ContainerStarted","Data":"28a84ae2975448b4c060bc39db415774c2073b817c2dfc94cc50c1ac7ad5d237"}
Nov 03 07:40:56.569158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:56.557112    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" podStartSLOduration=1.557091485 podStartE2EDuration="1.557091485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:40:56.554846204 +0000 UTC m=+2508.887215198" watchObservedRunningTime="2026-11-03 07:40:56.557091485 +0000 UTC m=+2508.889460480"
Nov 03 07:40:57.401923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:57.401868    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:40:57.403351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:57.403314    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 07:40:57.403458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:57.403385    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 07:40:58.403159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:58.402474    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 07:40:58.403159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:58.402538    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 07:40:59.399759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:59.399711    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 07:40:59.399979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:40:59.399778    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 07:41:00.399883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:00.399832    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 07:41:00.400442 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:00.399896    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 07:41:01.399869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:01.399842    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 07:41:01.399975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:01.399895    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 07:41:02.400569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:02.400529    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 07:41:02.401058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:02.400601    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 07:41:03.400392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.400354    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 07:41:03.400575 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.400421    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 07:41:03.729640 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.729555    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"]
Nov 03 07:41:03.733664 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.733642    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:03.849175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.849113    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:41:03.851911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.849535    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:41:03.851911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.849658    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:41:03.851911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.849766    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:41:03.852279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.851786    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"]
Nov 03 07:41:03.881941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.881546    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtbr\" (UniqueName: \"kubernetes.io/projected/1408b8a1-960c-425e-8457-0a528f1e7fe6-kube-api-access-phtbr\") pod \"maas-api-revoke-sub-0d14e0ee-bmxpr\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:03.881941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.881764    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d14e0ee-bmxpr\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:03.881941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.881797    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d14e0ee-bmxpr\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:03.983632 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.983541    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d14e0ee-bmxpr\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:03.983632 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.983598    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d14e0ee-bmxpr\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:03.983866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.983734    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phtbr\" (UniqueName: \"kubernetes.io/projected/1408b8a1-960c-425e-8457-0a528f1e7fe6-kube-api-access-phtbr\") pod \"maas-api-revoke-sub-0d14e0ee-bmxpr\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:03.984540 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.984512    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d14e0ee-bmxpr\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:03.987225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:03.987189    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d14e0ee-bmxpr\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:04.066463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:04.066424    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtbr\" (UniqueName: \"kubernetes.io/projected/1408b8a1-960c-425e-8457-0a528f1e7fe6-kube-api-access-phtbr\") pod \"maas-api-revoke-sub-0d14e0ee-bmxpr\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:04.350318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:04.350270    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:04.664415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:04.664304    4470 patch_prober.go:29] interesting pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:41:04.664415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:04.664381    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:41:04.751251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:04.751202    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"]
Nov 03 07:41:04.762951 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:41:04.762907    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:41:04.763824 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:41:04.763796    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1408b8a1_960c_425e_8457_0a528f1e7fe6.slice/crio-8d57092f312294a2361bc52a6a5b2654fa9012c8e87340ebbebd1fd38782187e WatchSource:0}: Error finding container 8d57092f312294a2361bc52a6a5b2654fa9012c8e87340ebbebd1fd38782187e: Status 404 returned error can't find the container with id 8d57092f312294a2361bc52a6a5b2654fa9012c8e87340ebbebd1fd38782187e
Nov 03 07:41:04.809034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:04.808997    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:41:04.809527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:04.809503    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:41:04.810338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:04.810319    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:41:05.556871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:05.556817    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr" event={"ID":"1408b8a1-960c-425e-8457-0a528f1e7fe6","Type":"ContainerStarted","Data":"849c833ea77ac8210ec576b2c1a8e9f20362f9ccfc65c0ad54b8759de0d09193"}
Nov 03 07:41:05.557070 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:05.556892    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr" event={"ID":"1408b8a1-960c-425e-8457-0a528f1e7fe6","Type":"ContainerStarted","Data":"8d57092f312294a2361bc52a6a5b2654fa9012c8e87340ebbebd1fd38782187e"}
Nov 03 07:41:05.558596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:05.558576    4470 generic.go:356] "Generic (PLEG): container finished" podID="d336591b-65cf-4a16-a050-569193c8f481" containerID="cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d" exitCode=0
Nov 03 07:41:05.558721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:05.558648    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" event={"ID":"d336591b-65cf-4a16-a050-569193c8f481","Type":"ContainerDied","Data":"cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d"}
Nov 03 07:41:05.559012 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:05.558999    4470 scope.go:122] "RemoveContainer" containerID="cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d"
Nov 03 07:41:05.560394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:05.560367    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:41:05.560474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:05.560432    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:41:05.560474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:05.560463    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:41:05.678291 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:05.678228    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr" podStartSLOduration=2.678212312 podStartE2EDuration="2.678212312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:41:05.676842951 +0000 UTC m=+2518.009211946" watchObservedRunningTime="2026-11-03 07:41:05.678212312 +0000 UTC m=+2518.010581339"
Nov 03 07:41:06.583872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:06.583840    4470 generic.go:356] "Generic (PLEG): container finished" podID="1408b8a1-960c-425e-8457-0a528f1e7fe6" containerID="849c833ea77ac8210ec576b2c1a8e9f20362f9ccfc65c0ad54b8759de0d09193" exitCode=0
Nov 03 07:41:06.584375 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:06.583926    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr" event={"ID":"1408b8a1-960c-425e-8457-0a528f1e7fe6","Type":"ContainerDied","Data":"849c833ea77ac8210ec576b2c1a8e9f20362f9ccfc65c0ad54b8759de0d09193"}
Nov 03 07:41:06.586002 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:06.585977    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" event={"ID":"d336591b-65cf-4a16-a050-569193c8f481","Type":"ContainerStarted","Data":"c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483"}
Nov 03 07:41:06.765583 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:06.765546    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:41:06.772316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:06.772279    4470 patch_prober.go:29] interesting pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Nov 03 07:41:06.772531 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:06.772334    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Nov 03 07:41:07.774347 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:07.774307    4470 patch_prober.go:29] interesting pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Nov 03 07:41:07.774883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:07.774411    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Nov 03 07:41:07.795295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:07.795263    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:07.939736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:07.939651    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1408b8a1-960c-425e-8457-0a528f1e7fe6-kube-api-access-phtbr\" (UniqueName: \"kubernetes.io/projected/1408b8a1-960c-425e-8457-0a528f1e7fe6-kube-api-access-phtbr\") pod \"1408b8a1-960c-425e-8457-0a528f1e7fe6\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") "
Nov 03 07:41:07.939908 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:07.939785    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-openshift-ca\") pod \"1408b8a1-960c-425e-8457-0a528f1e7fe6\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") "
Nov 03 07:41:07.940042 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:07.940023    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-xks-ca\") pod \"1408b8a1-960c-425e-8457-0a528f1e7fe6\" (UID: \"1408b8a1-960c-425e-8457-0a528f1e7fe6\") "
Nov 03 07:41:07.940180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:07.940123    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-openshift-ca" pod "1408b8a1-960c-425e-8457-0a528f1e7fe6" (UID: "1408b8a1-960c-425e-8457-0a528f1e7fe6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:41:07.940628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:07.940604    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:07.942764 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:07.942738    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1408b8a1-960c-425e-8457-0a528f1e7fe6-kube-api-access-phtbr" pod "1408b8a1-960c-425e-8457-0a528f1e7fe6" (UID: "1408b8a1-960c-425e-8457-0a528f1e7fe6"). InnerVolumeSpecName "kube-api-access-phtbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:41:07.944504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:07.944478    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-xks-ca" pod "1408b8a1-960c-425e-8457-0a528f1e7fe6" (UID: "1408b8a1-960c-425e-8457-0a528f1e7fe6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:41:08.042037 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:08.042000    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phtbr\" (UniqueName: \"kubernetes.io/projected/1408b8a1-960c-425e-8457-0a528f1e7fe6-kube-api-access-phtbr\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:08.042037 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:08.042030    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1408b8a1-960c-425e-8457-0a528f1e7fe6-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:08.606608 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:08.606579    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"
Nov 03 07:41:08.606790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:08.606608    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr" event={"ID":"1408b8a1-960c-425e-8457-0a528f1e7fe6","Type":"ContainerDied","Data":"8d57092f312294a2361bc52a6a5b2654fa9012c8e87340ebbebd1fd38782187e"}
Nov 03 07:41:08.606790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:08.606654    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d57092f312294a2361bc52a6a5b2654fa9012c8e87340ebbebd1fd38782187e"
Nov 03 07:41:08.767157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:08.766593    4470 patch_prober.go:29] interesting pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Nov 03 07:41:08.767157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:08.766653    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Nov 03 07:41:09.766161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:09.756550    4470 patch_prober.go:29] interesting pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Nov 03 07:41:09.766161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:09.756613    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Nov 03 07:41:10.759162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:10.752774    4470 patch_prober.go:29] interesting pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Nov 03 07:41:10.759162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:10.753066    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Nov 03 07:41:11.819834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:11.819798    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:41:11.821113 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:11.820611    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:41:11.821302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:11.821246    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:41:14.311883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:14.311852    4470 scope.go:122] "RemoveContainer" containerID="9d153c8ecd3dfcba6bb8baf4ea48e42dd4ea064f65b2e21a3b1f089c2479c08c"
Nov 03 07:41:14.321585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:14.321554    4470 scope.go:122] "RemoveContainer" containerID="1997a4525f9fc8b19f8136f5c967f614893a386597bbf1a8bab7d79d052e32ff"
Nov 03 07:41:14.330390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:14.330368    4470 scope.go:122] "RemoveContainer" containerID="e9376f10aa463e04a908122af72b574cef85e614a07fc007f911cccca68e147c"
Nov 03 07:41:14.907187 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:14.907153    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"]
Nov 03 07:41:14.907828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:14.907811    4470 cpu_manager.go:414] "containerMap: removing container" podUID="1408b8a1-960c-425e-8457-0a528f1e7fe6" containerName="revoke-keys"
Nov 03 07:41:14.907904 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:14.907830    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1408b8a1-960c-425e-8457-0a528f1e7fe6" containerName="revoke-keys"
Nov 03 07:41:14.907953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:14.907944    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="1408b8a1-960c-425e-8457-0a528f1e7fe6" containerName="revoke-keys"
Nov 03 07:41:14.911196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:14.911177    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:14.967841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:14.967811    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"]
Nov 03 07:41:15.014264 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.014222    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:41:15.014264 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.014229    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:41:15.014483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.014272    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:41:15.016494 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.016480    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:41:15.059144 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.059099    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4dv\" (UniqueName: \"kubernetes.io/projected/67ca14c0-1670-41a9-be4e-315e4d4bb03b-kube-api-access-lm4dv\") pod \"maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:15.059318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.059276    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:15.059318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.059305    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:15.161300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.161213    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:15.161300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.161257    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:15.161567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.161407    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4dv\" (UniqueName: \"kubernetes.io/projected/67ca14c0-1670-41a9-be4e-315e4d4bb03b-kube-api-access-lm4dv\") pod \"maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:15.161994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.161968    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:15.163705 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.163682    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:15.225803 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.225775    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4dv\" (UniqueName: \"kubernetes.io/projected/67ca14c0-1670-41a9-be4e-315e4d4bb03b-kube-api-access-lm4dv\") pod \"maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:15.524210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.524103    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:15.901507 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:15.901478    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"]
Nov 03 07:41:15.901893 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:41:15.901870    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ca14c0_1670_41a9_be4e_315e4d4bb03b.slice/crio-14766b4aa30a42a3f87265b1d46a4b592414a83811561b1134cb444d24318841 WatchSource:0}: Error finding container 14766b4aa30a42a3f87265b1d46a4b592414a83811561b1134cb444d24318841: Status 404 returned error can't find the container with id 14766b4aa30a42a3f87265b1d46a4b592414a83811561b1134cb444d24318841
Nov 03 07:41:16.662551 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:16.662510    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d" event={"ID":"67ca14c0-1670-41a9-be4e-315e4d4bb03b","Type":"ContainerStarted","Data":"7dd052153be9780ad848e86eabe6951f7639c73594b7203a25711785b9bbad9d"}
Nov 03 07:41:16.662940 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:16.662571    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d" event={"ID":"67ca14c0-1670-41a9-be4e-315e4d4bb03b","Type":"ContainerStarted","Data":"14766b4aa30a42a3f87265b1d46a4b592414a83811561b1134cb444d24318841"}
Nov 03 07:41:16.784931 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:16.784884    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d" podStartSLOduration=2.784868913 podStartE2EDuration="2.784868913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:41:16.781992363 +0000 UTC m=+2529.114361369" watchObservedRunningTime="2026-11-03 07:41:16.784868913 +0000 UTC m=+2529.117237907"
Nov 03 07:41:17.676698 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:17.676668    4470 generic.go:356] "Generic (PLEG): container finished" podID="67ca14c0-1670-41a9-be4e-315e4d4bb03b" containerID="7dd052153be9780ad848e86eabe6951f7639c73594b7203a25711785b9bbad9d" exitCode=0
Nov 03 07:41:17.677259 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:17.677239    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d" event={"ID":"67ca14c0-1670-41a9-be4e-315e4d4bb03b","Type":"ContainerDied","Data":"7dd052153be9780ad848e86eabe6951f7639c73594b7203a25711785b9bbad9d"}
Nov 03 07:41:18.892765 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:18.892742    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:19.031429 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.031334    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ca14c0-1670-41a9-be4e-315e4d4bb03b-kube-api-access-lm4dv\" (UniqueName: \"kubernetes.io/projected/67ca14c0-1670-41a9-be4e-315e4d4bb03b-kube-api-access-lm4dv\") pod \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") "
Nov 03 07:41:19.031612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.031529    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-xks-ca\") pod \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") "
Nov 03 07:41:19.031612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.031607    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-openshift-ca\") pod \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\" (UID: \"67ca14c0-1670-41a9-be4e-315e4d4bb03b\") "
Nov 03 07:41:19.031941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.031911    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-openshift-ca" pod "67ca14c0-1670-41a9-be4e-315e4d4bb03b" (UID: "67ca14c0-1670-41a9-be4e-315e4d4bb03b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:41:19.032215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.032197    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:19.033746 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.033725    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-xks-ca" pod "67ca14c0-1670-41a9-be4e-315e4d4bb03b" (UID: "67ca14c0-1670-41a9-be4e-315e4d4bb03b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:41:19.033986 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.033965    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ca14c0-1670-41a9-be4e-315e4d4bb03b-kube-api-access-lm4dv" pod "67ca14c0-1670-41a9-be4e-315e4d4bb03b" (UID: "67ca14c0-1670-41a9-be4e-315e4d4bb03b"). InnerVolumeSpecName "kube-api-access-lm4dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:41:19.133670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.133628    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm4dv\" (UniqueName: \"kubernetes.io/projected/67ca14c0-1670-41a9-be4e-315e4d4bb03b-kube-api-access-lm4dv\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:19.133670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.133676    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ca14c0-1670-41a9-be4e-315e4d4bb03b-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:19.699273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.699241    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"
Nov 03 07:41:19.699273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.699266    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d" event={"ID":"67ca14c0-1670-41a9-be4e-315e4d4bb03b","Type":"ContainerDied","Data":"14766b4aa30a42a3f87265b1d46a4b592414a83811561b1134cb444d24318841"}
Nov 03 07:41:19.699487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:19.699341    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14766b4aa30a42a3f87265b1d46a4b592414a83811561b1134cb444d24318841"
Nov 03 07:41:26.740434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:26.740375    4470 patch_prober.go:29] interesting pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:41:26.741119 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:26.741087    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:41:27.786998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:27.786969    4470 generic.go:356] "Generic (PLEG): container finished" podID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerID="c327cf187cec1f3259aca10154dc78afda76cd2bbd1110ee0e92519a1b2c21da" exitCode=0
Nov 03 07:41:27.787451 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:27.787038    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" event={"ID":"8605b360-6f53-41b3-a1ab-295c6bbdd046","Type":"ContainerDied","Data":"c327cf187cec1f3259aca10154dc78afda76cd2bbd1110ee0e92519a1b2c21da"}
Nov 03 07:41:27.787451 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:27.787397    4470 scope.go:122] "RemoveContainer" containerID="c327cf187cec1f3259aca10154dc78afda76cd2bbd1110ee0e92519a1b2c21da"
Nov 03 07:41:27.788864 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:27.788836    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:41:27.788933 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:27.788901    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:41:27.788975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:27.788930    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:41:28.796277 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:28.796239    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" event={"ID":"8605b360-6f53-41b3-a1ab-295c6bbdd046","Type":"ContainerStarted","Data":"92d25661d9056277ba97bfb36c8b9ef9ee2ee56eb850278fdbc53e3c2ac80c96"}
Nov 03 07:41:28.911693 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:28.911659    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"]
Nov 03 07:41:28.914702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:28.914678    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3b41b9-xjqbn"]
Nov 03 07:41:29.736752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:29.736722    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:41:29.736954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:29.736916    4470 patch_prober.go:29] interesting pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 07:41:29.737000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:29.736972    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 07:41:29.806726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:29.806693    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d477fb64-5ef6-4e61-977f-4477417704f6" path="/var/lib/kubelet/pods/d477fb64-5ef6-4e61-977f-4477417704f6/volumes"
Nov 03 07:41:30.737122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:30.737088    4470 patch_prober.go:29] interesting pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 07:41:30.737326 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:30.737179    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 07:41:31.737667 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:31.737628    4470 patch_prober.go:29] interesting pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 07:41:31.738072 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:31.737684    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 07:41:32.737179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:32.737116    4470 patch_prober.go:29] interesting pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 07:41:32.737359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:32.737198    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 07:41:33.737679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:33.737643    4470 patch_prober.go:29] interesting pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 07:41:33.738093 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:33.737719    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 07:41:34.741532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:34.741501    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:41:34.741977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:34.741886    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:41:34.742676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:34.742654    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:41:34.760127 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:41:34.760098    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:41:38.685937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.685905    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"]
Nov 03 07:41:38.686571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.686556    4470 cpu_manager.go:414] "containerMap: removing container" podUID="67ca14c0-1670-41a9-be4e-315e4d4bb03b" containerName="revoke-keys"
Nov 03 07:41:38.686616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.686572    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ca14c0-1670-41a9-be4e-315e4d4bb03b" containerName="revoke-keys"
Nov 03 07:41:38.686664 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.686655    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ca14c0-1670-41a9-be4e-315e4d4bb03b" containerName="revoke-keys"
Nov 03 07:41:38.689635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.689620    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:38.748384 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.748352    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"]
Nov 03 07:41:38.795272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.795226    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:41:38.795272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.795242    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:41:38.795272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.795260    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:41:38.795556 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.795330    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:41:38.798902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.798880    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"]
Nov 03 07:41:38.802689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.802671    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:38.829183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.829146    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cceb04-plw96\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:38.829327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.829246    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cceb04-plw96\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:38.829327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.829300    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djst6\" (UniqueName: \"kubernetes.io/projected/1629fa1b-2823-4b28-8c15-110b8487c13f-kube-api-access-djst6\") pod \"maas-api-revoke-sub-33cceb04-plw96\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:38.857284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.857249    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"]
Nov 03 07:41:38.931253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.931221    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxqzx\" (UniqueName: \"kubernetes.io/projected/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-kube-api-access-lxqzx\") pod \"maas-api-revoke-sub-ca6686df-xrdzj\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:38.931445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.931288    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca6686df-xrdzj\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:38.931445 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.931402    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca6686df-xrdzj\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:38.931560 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.931501    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cceb04-plw96\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:38.931629 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.931585    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cceb04-plw96\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:38.931695 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.931678    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djst6\" (UniqueName: \"kubernetes.io/projected/1629fa1b-2823-4b28-8c15-110b8487c13f-kube-api-access-djst6\") pod \"maas-api-revoke-sub-33cceb04-plw96\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:38.932188 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.932165    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cceb04-plw96\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:38.934148 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.934114    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cceb04-plw96\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:38.996686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:38.996604    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djst6\" (UniqueName: \"kubernetes.io/projected/1629fa1b-2823-4b28-8c15-110b8487c13f-kube-api-access-djst6\") pod \"maas-api-revoke-sub-33cceb04-plw96\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:39.003522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.003491    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:39.032692 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.032667    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxqzx\" (UniqueName: \"kubernetes.io/projected/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-kube-api-access-lxqzx\") pod \"maas-api-revoke-sub-ca6686df-xrdzj\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:39.032821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.032745    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca6686df-xrdzj\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:39.032821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.032805    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca6686df-xrdzj\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:39.033671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.033617    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca6686df-xrdzj\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:39.035730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.035696    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca6686df-xrdzj\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:39.097426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.097393    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxqzx\" (UniqueName: \"kubernetes.io/projected/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-kube-api-access-lxqzx\") pod \"maas-api-revoke-sub-ca6686df-xrdzj\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:39.117485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.117454    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:39.388638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.388609    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"]
Nov 03 07:41:39.388768 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:41:39.388749    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1629fa1b_2823_4b28_8c15_110b8487c13f.slice/crio-e86abb6143b4f0122c5673ce6fe89d6e595442c668f82741c289727b9388fbb1 WatchSource:0}: Error finding container e86abb6143b4f0122c5673ce6fe89d6e595442c668f82741c289727b9388fbb1: Status 404 returned error can't find the container with id e86abb6143b4f0122c5673ce6fe89d6e595442c668f82741c289727b9388fbb1
Nov 03 07:41:39.501293 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.501263    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"]
Nov 03 07:41:39.501952 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:41:39.501927    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba9ecb3_32ef_4617_a1f7_7004d4a43d1a.slice/crio-927411bc6146cde2bd3976efd5093b4a6d20909cc204971568e97c53c08f1942 WatchSource:0}: Error finding container 927411bc6146cde2bd3976efd5093b4a6d20909cc204971568e97c53c08f1942: Status 404 returned error can't find the container with id 927411bc6146cde2bd3976efd5093b4a6d20909cc204971568e97c53c08f1942
Nov 03 07:41:39.877205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.877173    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96" event={"ID":"1629fa1b-2823-4b28-8c15-110b8487c13f","Type":"ContainerStarted","Data":"c1651dc18d818820bfdbd3e20ef46661bb4108a2791155cde473369ced9696e4"}
Nov 03 07:41:39.877677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.877233    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96" event={"ID":"1629fa1b-2823-4b28-8c15-110b8487c13f","Type":"ContainerStarted","Data":"e86abb6143b4f0122c5673ce6fe89d6e595442c668f82741c289727b9388fbb1"}
Nov 03 07:41:39.878959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.878934    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj" event={"ID":"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a","Type":"ContainerStarted","Data":"36ab4483aee3cafdf2c3eb664fddcf608a6877bb5f3d099857843105e9fbb912"}
Nov 03 07:41:39.879094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.878973    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj" event={"ID":"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a","Type":"ContainerStarted","Data":"927411bc6146cde2bd3976efd5093b4a6d20909cc204971568e97c53c08f1942"}
Nov 03 07:41:39.998003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:39.997898    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96" podStartSLOduration=1.997885462 podStartE2EDuration="1.997885462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:41:39.995665218 +0000 UTC m=+2552.328034218" watchObservedRunningTime="2026-11-03 07:41:39.997885462 +0000 UTC m=+2552.330254455"
Nov 03 07:41:40.115923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:40.115871    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj" podStartSLOduration=2.115855387 podStartE2EDuration="2.115855387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:41:40.115049932 +0000 UTC m=+2552.447418926" watchObservedRunningTime="2026-11-03 07:41:40.115855387 +0000 UTC m=+2552.448224382"
Nov 03 07:41:40.887478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:40.887445    4470 generic.go:356] "Generic (PLEG): container finished" podID="3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a" containerID="36ab4483aee3cafdf2c3eb664fddcf608a6877bb5f3d099857843105e9fbb912" exitCode=0
Nov 03 07:41:40.887902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:40.887538    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj" event={"ID":"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a","Type":"ContainerDied","Data":"36ab4483aee3cafdf2c3eb664fddcf608a6877bb5f3d099857843105e9fbb912"}
Nov 03 07:41:40.889247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:40.889227    4470 generic.go:356] "Generic (PLEG): container finished" podID="1629fa1b-2823-4b28-8c15-110b8487c13f" containerID="c1651dc18d818820bfdbd3e20ef46661bb4108a2791155cde473369ced9696e4" exitCode=0
Nov 03 07:41:40.889372 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:40.889254    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96" event={"ID":"1629fa1b-2823-4b28-8c15-110b8487c13f","Type":"ContainerDied","Data":"c1651dc18d818820bfdbd3e20ef46661bb4108a2791155cde473369ced9696e4"}
Nov 03 07:41:42.098017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.097992    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:42.101384 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.101363    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:42.174482 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.174401    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-kube-api-access-lxqzx\" (UniqueName: \"kubernetes.io/projected/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-kube-api-access-lxqzx\") pod \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") "
Nov 03 07:41:42.174642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.174492    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-openshift-ca\") pod \"1629fa1b-2823-4b28-8c15-110b8487c13f\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") "
Nov 03 07:41:42.174642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.174545    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1629fa1b-2823-4b28-8c15-110b8487c13f-kube-api-access-djst6\" (UniqueName: \"kubernetes.io/projected/1629fa1b-2823-4b28-8c15-110b8487c13f-kube-api-access-djst6\") pod \"1629fa1b-2823-4b28-8c15-110b8487c13f\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") "
Nov 03 07:41:42.174642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.174565    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-xks-ca\") pod \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") "
Nov 03 07:41:42.174642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.174597    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-openshift-ca\") pod \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\" (UID: \"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a\") "
Nov 03 07:41:42.174642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.174634    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-xks-ca\") pod \"1629fa1b-2823-4b28-8c15-110b8487c13f\" (UID: \"1629fa1b-2823-4b28-8c15-110b8487c13f\") "
Nov 03 07:41:42.174915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.174880    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-openshift-ca" pod "1629fa1b-2823-4b28-8c15-110b8487c13f" (UID: "1629fa1b-2823-4b28-8c15-110b8487c13f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:41:42.175005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.174978    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-openshift-ca" pod "3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a" (UID: "3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:41:42.175810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.175782    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:42.175962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.175814    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:42.177428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.177379    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-xks-ca" pod "3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a" (UID: "3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:41:42.177428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.177407    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-kube-api-access-lxqzx" pod "3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a" (UID: "3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a"). InnerVolumeSpecName "kube-api-access-lxqzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:41:42.177604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.177492    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1629fa1b-2823-4b28-8c15-110b8487c13f-kube-api-access-djst6" pod "1629fa1b-2823-4b28-8c15-110b8487c13f" (UID: "1629fa1b-2823-4b28-8c15-110b8487c13f"). InnerVolumeSpecName "kube-api-access-djst6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:41:42.177604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.177575    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-xks-ca" pod "1629fa1b-2823-4b28-8c15-110b8487c13f" (UID: "1629fa1b-2823-4b28-8c15-110b8487c13f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:41:42.276601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.276567    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxqzx\" (UniqueName: \"kubernetes.io/projected/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-kube-api-access-lxqzx\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:42.276601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.276597    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djst6\" (UniqueName: \"kubernetes.io/projected/1629fa1b-2823-4b28-8c15-110b8487c13f-kube-api-access-djst6\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:42.276601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.276607    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:42.276835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.276616    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1629fa1b-2823-4b28-8c15-110b8487c13f-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:42.904633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.904599    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"
Nov 03 07:41:42.904633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.904619    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96" event={"ID":"1629fa1b-2823-4b28-8c15-110b8487c13f","Type":"ContainerDied","Data":"e86abb6143b4f0122c5673ce6fe89d6e595442c668f82741c289727b9388fbb1"}
Nov 03 07:41:42.904858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.904673    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e86abb6143b4f0122c5673ce6fe89d6e595442c668f82741c289727b9388fbb1"
Nov 03 07:41:42.906015 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.905998    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"
Nov 03 07:41:42.906166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.906019    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj" event={"ID":"3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a","Type":"ContainerDied","Data":"927411bc6146cde2bd3976efd5093b4a6d20909cc204971568e97c53c08f1942"}
Nov 03 07:41:42.906166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:42.906055    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="927411bc6146cde2bd3976efd5093b4a6d20909cc204971568e97c53c08f1942"
Nov 03 07:41:55.647216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.646514    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"]
Nov 03 07:41:55.647686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.647467    4470 cpu_manager.go:414] "containerMap: removing container" podUID="1629fa1b-2823-4b28-8c15-110b8487c13f" containerName="revoke-keys"
Nov 03 07:41:55.647686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.647489    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1629fa1b-2823-4b28-8c15-110b8487c13f" containerName="revoke-keys"
Nov 03 07:41:55.647686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.647531    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a" containerName="revoke-keys"
Nov 03 07:41:55.647686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.647540    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a" containerName="revoke-keys"
Nov 03 07:41:55.647978 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.647705    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="1629fa1b-2823-4b28-8c15-110b8487c13f" containerName="revoke-keys"
Nov 03 07:41:55.647978 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.647728    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a" containerName="revoke-keys"
Nov 03 07:41:55.653273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.653250    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:55.759159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.758706    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:41:55.759159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.758855    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:41:55.759159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.758924    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:41:55.823163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.822691    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:41:55.932894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.932801    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:55.933198 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.933173    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvk9p\" (UniqueName: \"kubernetes.io/projected/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-kube-api-access-nvk9p\") pod \"maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:55.942313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:55.933504    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:56.055100 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:56.055060    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:56.055654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:56.055626    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:56.055834 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:56.055815    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvk9p\" (UniqueName: \"kubernetes.io/projected/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-kube-api-access-nvk9p\") pod \"maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:56.083187 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:56.072839    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"]
Nov 03 07:41:56.107160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:56.085429    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:56.107422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:56.087915    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:56.163925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:56.163891    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvk9p\" (UniqueName: \"kubernetes.io/projected/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-kube-api-access-nvk9p\") pod \"maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:56.345155 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:56.342397    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:57.014920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:57.014883    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"]
Nov 03 07:41:57.208159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:57.207927    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq" event={"ID":"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc","Type":"ContainerStarted","Data":"ecb65f29e03c71d2d348c8c5ec71ba84596620d5d85ecacd35ab15999509df42"}
Nov 03 07:41:58.222437 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:58.222405    4470 generic.go:356] "Generic (PLEG): container finished" podID="d0e4d65c-7d82-44bd-abe3-c60b05bd89fc" containerID="9736ada0b8b121b8fa43a34a0290c07a83141fc331f79526af250861f52fc577" exitCode=0
Nov 03 07:41:58.222963 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:58.222478    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq" event={"ID":"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc","Type":"ContainerDied","Data":"9736ada0b8b121b8fa43a34a0290c07a83141fc331f79526af250861f52fc577"}
Nov 03 07:41:59.397558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:59.397532    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:41:59.470486 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:59.470447    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-kube-api-access-nvk9p\" (UniqueName: \"kubernetes.io/projected/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-kube-api-access-nvk9p\") pod \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") "
Nov 03 07:41:59.470678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:59.470570    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-xks-ca\") pod \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") "
Nov 03 07:41:59.470678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:59.470642    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-openshift-ca\") pod \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\" (UID: \"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc\") "
Nov 03 07:41:59.471049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:59.471015    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-openshift-ca" pod "d0e4d65c-7d82-44bd-abe3-c60b05bd89fc" (UID: "d0e4d65c-7d82-44bd-abe3-c60b05bd89fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:41:59.471242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:59.471222    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:59.472808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:59.472785    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-xks-ca" pod "d0e4d65c-7d82-44bd-abe3-c60b05bd89fc" (UID: "d0e4d65c-7d82-44bd-abe3-c60b05bd89fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:41:59.472903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:59.472809    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-kube-api-access-nvk9p" pod "d0e4d65c-7d82-44bd-abe3-c60b05bd89fc" (UID: "d0e4d65c-7d82-44bd-abe3-c60b05bd89fc"). InnerVolumeSpecName "kube-api-access-nvk9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:41:59.572774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:59.572738    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvk9p\" (UniqueName: \"kubernetes.io/projected/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-kube-api-access-nvk9p\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:41:59.572774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:41:59.572770    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:00.239207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:00.239087    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"
Nov 03 07:42:00.239207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:00.239100    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq" event={"ID":"d0e4d65c-7d82-44bd-abe3-c60b05bd89fc","Type":"ContainerDied","Data":"ecb65f29e03c71d2d348c8c5ec71ba84596620d5d85ecacd35ab15999509df42"}
Nov 03 07:42:00.239207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:00.239190    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecb65f29e03c71d2d348c8c5ec71ba84596620d5d85ecacd35ab15999509df42"
Nov 03 07:42:04.760665 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:42:04.760626    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:42:06.333367 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.333334    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"]
Nov 03 07:42:06.333913 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.333900    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d0e4d65c-7d82-44bd-abe3-c60b05bd89fc" containerName="revoke-keys"
Nov 03 07:42:06.333958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.333915    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e4d65c-7d82-44bd-abe3-c60b05bd89fc" containerName="revoke-keys"
Nov 03 07:42:06.334027 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.334018    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e4d65c-7d82-44bd-abe3-c60b05bd89fc" containerName="revoke-keys"
Nov 03 07:42:06.337977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.337957    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:06.396903 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.396872    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"]
Nov 03 07:42:06.442808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.442772    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:42:06.442808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.442772    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:42:06.443057 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.442821    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:42:06.443057 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.442945    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:42:06.466226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.466189    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnfr\" (UniqueName: \"kubernetes.io/projected/14811088-b434-498a-8231-707393e1d590-kube-api-access-sbnfr\") pod \"maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft\" (UID: \"14811088-b434-498a-8231-707393e1d590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:06.466381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.466226    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14811088-b434-498a-8231-707393e1d590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft\" (UID: \"14811088-b434-498a-8231-707393e1d590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:06.466381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.466344    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14811088-b434-498a-8231-707393e1d590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft\" (UID: \"14811088-b434-498a-8231-707393e1d590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:06.567430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.567386    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnfr\" (UniqueName: \"kubernetes.io/projected/14811088-b434-498a-8231-707393e1d590-kube-api-access-sbnfr\") pod \"maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft\" (UID: \"14811088-b434-498a-8231-707393e1d590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:06.567430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.567429    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14811088-b434-498a-8231-707393e1d590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft\" (UID: \"14811088-b434-498a-8231-707393e1d590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:06.567717 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.567467    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14811088-b434-498a-8231-707393e1d590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft\" (UID: \"14811088-b434-498a-8231-707393e1d590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:06.568230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.568197    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14811088-b434-498a-8231-707393e1d590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft\" (UID: \"14811088-b434-498a-8231-707393e1d590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:06.570185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.570164    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14811088-b434-498a-8231-707393e1d590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft\" (UID: \"14811088-b434-498a-8231-707393e1d590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:06.631262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.631193    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnfr\" (UniqueName: \"kubernetes.io/projected/14811088-b434-498a-8231-707393e1d590-kube-api-access-sbnfr\") pod \"maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft\" (UID: \"14811088-b434-498a-8231-707393e1d590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:06.651052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:06.651022    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:07.031459 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:42:07.031420    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14811088_b434_498a_8231_707393e1d590.slice/crio-7af4dff592d82e51d84f046ddeff249a362100d5af1acca495999a194a05cdbc WatchSource:0}: Error finding container 7af4dff592d82e51d84f046ddeff249a362100d5af1acca495999a194a05cdbc: Status 404 returned error can't find the container with id 7af4dff592d82e51d84f046ddeff249a362100d5af1acca495999a194a05cdbc
Nov 03 07:42:07.031585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:07.031491    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"]
Nov 03 07:42:07.301369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:07.301270    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft" event={"ID":"14811088-b434-498a-8231-707393e1d590","Type":"ContainerStarted","Data":"7af4dff592d82e51d84f046ddeff249a362100d5af1acca495999a194a05cdbc"}
Nov 03 07:42:08.310800 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:08.310718    4470 generic.go:356] "Generic (PLEG): container finished" podID="14811088-b434-498a-8231-707393e1d590" containerID="bc660400b7f3dfa465fab2ac50dc541ceb4b159c616863119db2f5772677c997" exitCode=0
Nov 03 07:42:08.310800 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:08.310772    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft" event={"ID":"14811088-b434-498a-8231-707393e1d590","Type":"ContainerDied","Data":"bc660400b7f3dfa465fab2ac50dc541ceb4b159c616863119db2f5772677c997"}
Nov 03 07:42:09.471029 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:09.471003    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:09.627376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:09.627304    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14811088-b434-498a-8231-707393e1d590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14811088-b434-498a-8231-707393e1d590-maas-api-xks-ca\") pod \"14811088-b434-498a-8231-707393e1d590\" (UID: \"14811088-b434-498a-8231-707393e1d590\") "
Nov 03 07:42:09.627526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:09.627512    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14811088-b434-498a-8231-707393e1d590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14811088-b434-498a-8231-707393e1d590-maas-api-openshift-ca\") pod \"14811088-b434-498a-8231-707393e1d590\" (UID: \"14811088-b434-498a-8231-707393e1d590\") "
Nov 03 07:42:09.627597 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:09.627573    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14811088-b434-498a-8231-707393e1d590-kube-api-access-sbnfr\" (UniqueName: \"kubernetes.io/projected/14811088-b434-498a-8231-707393e1d590-kube-api-access-sbnfr\") pod \"14811088-b434-498a-8231-707393e1d590\" (UID: \"14811088-b434-498a-8231-707393e1d590\") "
Nov 03 07:42:09.627940 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:09.627912    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14811088-b434-498a-8231-707393e1d590-maas-api-openshift-ca" pod "14811088-b434-498a-8231-707393e1d590" (UID: "14811088-b434-498a-8231-707393e1d590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:42:09.628297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:09.628277    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14811088-b434-498a-8231-707393e1d590-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:09.630030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:09.629996    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14811088-b434-498a-8231-707393e1d590-kube-api-access-sbnfr" pod "14811088-b434-498a-8231-707393e1d590" (UID: "14811088-b434-498a-8231-707393e1d590"). InnerVolumeSpecName "kube-api-access-sbnfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:42:09.630391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:09.630359    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14811088-b434-498a-8231-707393e1d590-maas-api-xks-ca" pod "14811088-b434-498a-8231-707393e1d590" (UID: "14811088-b434-498a-8231-707393e1d590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:42:09.729592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:09.729550    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbnfr\" (UniqueName: \"kubernetes.io/projected/14811088-b434-498a-8231-707393e1d590-kube-api-access-sbnfr\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:09.729592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:09.729590    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14811088-b434-498a-8231-707393e1d590-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:10.342837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:10.342804    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"
Nov 03 07:42:10.343041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:10.342804    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft" event={"ID":"14811088-b434-498a-8231-707393e1d590","Type":"ContainerDied","Data":"7af4dff592d82e51d84f046ddeff249a362100d5af1acca495999a194a05cdbc"}
Nov 03 07:42:10.343041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:10.342954    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7af4dff592d82e51d84f046ddeff249a362100d5af1acca495999a194a05cdbc"
Nov 03 07:42:14.371375 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:14.371340    4470 scope.go:122] "RemoveContainer" containerID="b19711f889fbc8253ce23ab97cc8037d1884188ba27386e2657506f4d7a1a23e"
Nov 03 07:42:14.381426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:14.381404    4470 scope.go:122] "RemoveContainer" containerID="26f9a6a4e68b1734a9ac7568214acd030e7e002363f7710227547efcd5b50e2d"
Nov 03 07:42:14.390656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:14.390627    4470 scope.go:122] "RemoveContainer" containerID="a040bf53e0f88c1ed1a552026bbff9f9e3f2b62c6e918756c1bc35e36a369f28"
Nov 03 07:42:14.403581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:14.403551    4470 scope.go:122] "RemoveContainer" containerID="2b1a62896a0782d1ef02a46126aec2fe83b4d880facad0a9f955ad700542d2f6"
Nov 03 07:42:14.413576 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:14.413554    4470 scope.go:122] "RemoveContainer" containerID="8edfe2ade876f5d47e87438b43148d1b2e90f96e0a16876c63528c1272c803e0"
Nov 03 07:42:33.224029 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:33.223996    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"]
Nov 03 07:42:33.224707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:33.224277    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy" containerID="cri-o://3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f" gracePeriod=30
Nov 03 07:42:33.920696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:33.920662    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"]
Nov 03 07:42:33.920886 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:33.920864    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy" containerID="cri-o://9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af" gracePeriod=30
Nov 03 07:42:34.760357 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:42:34.760318    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:42:36.190513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:36.190468    4470 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:42:36.190924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:36.190539    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:42:38.500962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.500900    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:42:38.554102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.554068    4470 generic.go:356] "Generic (PLEG): container finished" podID="36405c56-94e9-4818-a713-ae9a8a677975" containerID="3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f" exitCode=0
Nov 03 07:42:38.554354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.554124    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"
Nov 03 07:42:38.554354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.554127    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" event={"ID":"36405c56-94e9-4818-a713-ae9a8a677975","Type":"ContainerDied","Data":"3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f"}
Nov 03 07:42:38.554354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.554287    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g" event={"ID":"36405c56-94e9-4818-a713-ae9a8a677975","Type":"ContainerDied","Data":"d791df004d5cacfa26194bd87da030427d51e9228eaaeb442f6f2b65bbf5acef"}
Nov 03 07:42:38.554354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.554319    4470 scope.go:122] "RemoveContainer" containerID="3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f"
Nov 03 07:42:38.566965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.566946    4470 scope.go:122] "RemoveContainer" containerID="0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7"
Nov 03 07:42:38.576553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.576534    4470 scope.go:122] "RemoveContainer" containerID="3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f"
Nov 03 07:42:38.576864 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:42:38.576843    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f\": container with ID starting with 3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f not found: ID does not exist" containerID="3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f"
Nov 03 07:42:38.576941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.576873    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f"} err="failed to get container status \"3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f\": rpc error: code = NotFound desc = could not find container \"3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f\": container with ID starting with 3418828884aaeda020b910a59d1350d164a5f27ffa9e56f04537afcdb60e7a2f not found: ID does not exist"
Nov 03 07:42:38.576941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.576890    4470 scope.go:122] "RemoveContainer" containerID="0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7"
Nov 03 07:42:38.577155 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:42:38.577118    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7\": container with ID starting with 0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7 not found: ID does not exist" containerID="0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7"
Nov 03 07:42:38.577220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.577167    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7"} err="failed to get container status \"0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7\": rpc error: code = NotFound desc = could not find container \"0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7\": container with ID starting with 0060da0e55a2262d5ea948441eb79a769fb3bb571d457e53d3d1c051863c30c7 not found: ID does not exist"
Nov 03 07:42:38.618313 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618279    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-certs\") pod \"36405c56-94e9-4818-a713-ae9a8a677975\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") "
Nov 03 07:42:38.618504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618356    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-socket\") pod \"36405c56-94e9-4818-a713-ae9a8a677975\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") "
Nov 03 07:42:38.618504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618459    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36405c56-94e9-4818-a713-ae9a8a677975-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36405c56-94e9-4818-a713-ae9a8a677975-istio-podinfo\") pod \"36405c56-94e9-4818-a713-ae9a8a677975\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") "
Nov 03 07:42:38.618639 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618517    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-credential-socket\") pod \"36405c56-94e9-4818-a713-ae9a8a677975\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") "
Nov 03 07:42:38.618639 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618581    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36405c56-94e9-4818-a713-ae9a8a677975-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36405c56-94e9-4818-a713-ae9a8a677975-istiod-ca-cert\") pod \"36405c56-94e9-4818-a713-ae9a8a677975\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") "
Nov 03 07:42:38.618639 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618577    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-certs" pod "36405c56-94e9-4818-a713-ae9a8a677975" (UID: "36405c56-94e9-4818-a713-ae9a8a677975"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:38.618639 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618622    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-kube-api-access-7m5hj\" (UniqueName: \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-kube-api-access-7m5hj\") pod \"36405c56-94e9-4818-a713-ae9a8a677975\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") "
Nov 03 07:42:38.618870 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618650    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-envoy\") pod \"36405c56-94e9-4818-a713-ae9a8a677975\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") "
Nov 03 07:42:38.618870 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618672    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-socket" pod "36405c56-94e9-4818-a713-ae9a8a677975" (UID: "36405c56-94e9-4818-a713-ae9a8a677975"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:38.618870 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618748    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-istio-token\" (UniqueName: \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-istio-token\") pod \"36405c56-94e9-4818-a713-ae9a8a677975\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") "
Nov 03 07:42:38.618870 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618794    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-data\") pod \"36405c56-94e9-4818-a713-ae9a8a677975\" (UID: \"36405c56-94e9-4818-a713-ae9a8a677975\") "
Nov 03 07:42:38.618870 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618829    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-credential-socket" pod "36405c56-94e9-4818-a713-ae9a8a677975" (UID: "36405c56-94e9-4818-a713-ae9a8a677975"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:38.619265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.618927    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36405c56-94e9-4818-a713-ae9a8a677975-istiod-ca-cert" pod "36405c56-94e9-4818-a713-ae9a8a677975" (UID: "36405c56-94e9-4818-a713-ae9a8a677975"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:42:38.619493 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.619455    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-data" pod "36405c56-94e9-4818-a713-ae9a8a677975" (UID: "36405c56-94e9-4818-a713-ae9a8a677975"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:38.619581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.619493    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:38.619581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.619519    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36405c56-94e9-4818-a713-ae9a8a677975-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:38.619581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.619564    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:38.619581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.619578    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:38.620954 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.620925    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36405c56-94e9-4818-a713-ae9a8a677975-istio-podinfo" pod "36405c56-94e9-4818-a713-ae9a8a677975" (UID: "36405c56-94e9-4818-a713-ae9a8a677975"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:42:38.621044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.620980    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-envoy" pod "36405c56-94e9-4818-a713-ae9a8a677975" (UID: "36405c56-94e9-4818-a713-ae9a8a677975"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:38.621442 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.621418    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-kube-api-access-7m5hj" pod "36405c56-94e9-4818-a713-ae9a8a677975" (UID: "36405c56-94e9-4818-a713-ae9a8a677975"). InnerVolumeSpecName "kube-api-access-7m5hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:42:38.621537 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.621439    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-istio-token" pod "36405c56-94e9-4818-a713-ae9a8a677975" (UID: "36405c56-94e9-4818-a713-ae9a8a677975"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:42:38.720460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.720421    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36405c56-94e9-4818-a713-ae9a8a677975-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:38.720460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.720454    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m5hj\" (UniqueName: \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-kube-api-access-7m5hj\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:38.720460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.720467    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:38.720706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.720475    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36405c56-94e9-4818-a713-ae9a8a677975-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:38.720706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.720483    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36405c56-94e9-4818-a713-ae9a8a677975-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:38.981709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.981673    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"]
Nov 03 07:42:38.987526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:38.987500    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g"]
Nov 03 07:42:39.168973 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.168947    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:42:39.328772 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.328736    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-kube-api-access-zpmxf\" (UniqueName: \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-kube-api-access-zpmxf\") pod \"3a557dbf-9da0-42bf-9354-38f46e3b9312\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") "
Nov 03 07:42:39.328980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.328785    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-token\" (UniqueName: \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-token\") pod \"3a557dbf-9da0-42bf-9354-38f46e3b9312\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") "
Nov 03 07:42:39.328980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.328839    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-credential-socket\") pod \"3a557dbf-9da0-42bf-9354-38f46e3b9312\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") "
Nov 03 07:42:39.328980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.328868    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-envoy\") pod \"3a557dbf-9da0-42bf-9354-38f46e3b9312\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") "
Nov 03 07:42:39.328980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.328927    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-podinfo\") pod \"3a557dbf-9da0-42bf-9354-38f46e3b9312\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") "
Nov 03 07:42:39.328980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.328956    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-certs\") pod \"3a557dbf-9da0-42bf-9354-38f46e3b9312\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") "
Nov 03 07:42:39.329311 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.329044    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-socket\") pod \"3a557dbf-9da0-42bf-9354-38f46e3b9312\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") "
Nov 03 07:42:39.329311 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.329097    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-data\") pod \"3a557dbf-9da0-42bf-9354-38f46e3b9312\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") "
Nov 03 07:42:39.329311 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.329190    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a557dbf-9da0-42bf-9354-38f46e3b9312-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a557dbf-9da0-42bf-9354-38f46e3b9312-istiod-ca-cert\") pod \"3a557dbf-9da0-42bf-9354-38f46e3b9312\" (UID: \"3a557dbf-9da0-42bf-9354-38f46e3b9312\") "
Nov 03 07:42:39.329311 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.329212    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-credential-socket" pod "3a557dbf-9da0-42bf-9354-38f46e3b9312" (UID: "3a557dbf-9da0-42bf-9354-38f46e3b9312"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:39.329531 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.329429    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-certs" pod "3a557dbf-9da0-42bf-9354-38f46e3b9312" (UID: "3a557dbf-9da0-42bf-9354-38f46e3b9312"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:39.329531 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.329511    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a557dbf-9da0-42bf-9354-38f46e3b9312-istiod-ca-cert" pod "3a557dbf-9da0-42bf-9354-38f46e3b9312" (UID: "3a557dbf-9da0-42bf-9354-38f46e3b9312"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:42:39.329645 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.329609    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-socket" pod "3a557dbf-9da0-42bf-9354-38f46e3b9312" (UID: "3a557dbf-9da0-42bf-9354-38f46e3b9312"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:39.330090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.329783    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-data" pod "3a557dbf-9da0-42bf-9354-38f46e3b9312" (UID: "3a557dbf-9da0-42bf-9354-38f46e3b9312"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:39.330090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.330022    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:39.330090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.330039    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a557dbf-9da0-42bf-9354-38f46e3b9312-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:39.330090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.330052    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:39.330090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.330060    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:39.330090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.330068    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:39.331837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.331809    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-envoy" pod "3a557dbf-9da0-42bf-9354-38f46e3b9312" (UID: "3a557dbf-9da0-42bf-9354-38f46e3b9312"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:39.331960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.331858    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-kube-api-access-zpmxf" pod "3a557dbf-9da0-42bf-9354-38f46e3b9312" (UID: "3a557dbf-9da0-42bf-9354-38f46e3b9312"). InnerVolumeSpecName "kube-api-access-zpmxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:42:39.332022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.331968    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-token" pod "3a557dbf-9da0-42bf-9354-38f46e3b9312" (UID: "3a557dbf-9da0-42bf-9354-38f46e3b9312"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:42:39.332074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.332051    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-podinfo" pod "3a557dbf-9da0-42bf-9354-38f46e3b9312" (UID: "3a557dbf-9da0-42bf-9354-38f46e3b9312"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:42:39.431071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.430993    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:39.431071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.431020    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:39.431071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.431029    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a557dbf-9da0-42bf-9354-38f46e3b9312-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:39.431071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.431039    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpmxf\" (UniqueName: \"kubernetes.io/projected/3a557dbf-9da0-42bf-9354-38f46e3b9312-kube-api-access-zpmxf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:39.563464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.563432    4470 generic.go:356] "Generic (PLEG): container finished" podID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerID="9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af" exitCode=0
Nov 03 07:42:39.563930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.563487    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"
Nov 03 07:42:39.563930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.563491    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" event={"ID":"3a557dbf-9da0-42bf-9354-38f46e3b9312","Type":"ContainerDied","Data":"9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af"}
Nov 03 07:42:39.563930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.563609    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55" event={"ID":"3a557dbf-9da0-42bf-9354-38f46e3b9312","Type":"ContainerDied","Data":"41133708dcf62077bb1650374f939628fc5f9bc872558d6cc9a0037532d43fc5"}
Nov 03 07:42:39.563930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.563632    4470 scope.go:122] "RemoveContainer" containerID="9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af"
Nov 03 07:42:39.575179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.575127    4470 scope.go:122] "RemoveContainer" containerID="624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f"
Nov 03 07:42:39.584795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.584773    4470 scope.go:122] "RemoveContainer" containerID="9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af"
Nov 03 07:42:39.585117 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:42:39.585088    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af\": container with ID starting with 9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af not found: ID does not exist" containerID="9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af"
Nov 03 07:42:39.585240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.585128    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af"} err="failed to get container status \"9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af\": rpc error: code = NotFound desc = could not find container \"9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af\": container with ID starting with 9073190211ca839fe2cbe32db1f6a85bfafd00100ec22cedd7761b369e2292af not found: ID does not exist"
Nov 03 07:42:39.585240 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.585188    4470 scope.go:122] "RemoveContainer" containerID="624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f"
Nov 03 07:42:39.585458 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:42:39.585439    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f\": container with ID starting with 624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f not found: ID does not exist" containerID="624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f"
Nov 03 07:42:39.585498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.585464    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f"} err="failed to get container status \"624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f\": rpc error: code = NotFound desc = could not find container \"624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f\": container with ID starting with 624aba9fcbcd88cef587f56f814b49bc14bde9c74f15b8bf5e925fe5fcc65f7f not found: ID does not exist"
Nov 03 07:42:39.737175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.737073    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"]
Nov 03 07:42:39.744981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.744954    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55"]
Nov 03 07:42:39.807619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.807583    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36405c56-94e9-4818-a713-ae9a8a677975" path="/var/lib/kubelet/pods/36405c56-94e9-4818-a713-ae9a8a677975/volumes"
Nov 03 07:42:39.808088 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:39.808073    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" path="/var/lib/kubelet/pods/3a557dbf-9da0-42bf-9354-38f46e3b9312/volumes"
Nov 03 07:42:42.967842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:42.967801    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"]
Nov 03 07:42:43.017822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:43.017780    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9697f7-z5s2d"]
Nov 03 07:42:43.806872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:43.806840    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ca14c0-1670-41a9-be4e-315e4d4bb03b" path="/var/lib/kubelet/pods/67ca14c0-1670-41a9-be4e-315e4d4bb03b/volumes"
Nov 03 07:42:50.417586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:50.417553    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"]
Nov 03 07:42:50.418231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:50.417765    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy" containerID="cri-o://13e55407d1b8fc3cc16416ad64065a60fa1c9f115112dae1d3f161a9281e5d21" gracePeriod=30
Nov 03 07:42:55.671808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.671771    4470 generic.go:356] "Generic (PLEG): container finished" podID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerID="13e55407d1b8fc3cc16416ad64065a60fa1c9f115112dae1d3f161a9281e5d21" exitCode=0
Nov 03 07:42:55.672210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.671833    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" event={"ID":"a05b8fa9-f2cf-4809-8bea-47a84600c2b7","Type":"ContainerDied","Data":"13e55407d1b8fc3cc16416ad64065a60fa1c9f115112dae1d3f161a9281e5d21"}
Nov 03 07:42:55.672210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.671920    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6" event={"ID":"a05b8fa9-f2cf-4809-8bea-47a84600c2b7","Type":"ContainerDied","Data":"b3c307adce47793272fc341ddf1ab5f9a7685f6cb1cc8f4cc2f3be96d18bfd65"}
Nov 03 07:42:55.672210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.671939    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3c307adce47793272fc341ddf1ab5f9a7685f6cb1cc8f4cc2f3be96d18bfd65"
Nov 03 07:42:55.679478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.679456    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:42:55.826856 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.826829    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istiod-ca-cert\") pod \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") "
Nov 03 07:42:55.827073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.826881    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-envoy\") pod \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") "
Nov 03 07:42:55.827073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.826915    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-podinfo\") pod \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") "
Nov 03 07:42:55.827073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.826935    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-certs\") pod \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") "
Nov 03 07:42:55.827073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.826956    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-socket\") pod \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") "
Nov 03 07:42:55.827073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.827028    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-kube-api-access-6x7cx\" (UniqueName: \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-kube-api-access-6x7cx\") pod \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") "
Nov 03 07:42:55.827335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.827169    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-credential-socket\") pod \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") "
Nov 03 07:42:55.827335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.827262    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istiod-ca-cert" pod "a05b8fa9-f2cf-4809-8bea-47a84600c2b7" (UID: "a05b8fa9-f2cf-4809-8bea-47a84600c2b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:42:55.827335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.827277    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-data\") pod \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") "
Nov 03 07:42:55.827517 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.827337    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-token\" (UniqueName: \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-token\") pod \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\" (UID: \"a05b8fa9-f2cf-4809-8bea-47a84600c2b7\") "
Nov 03 07:42:55.827517 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.827503    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-data" pod "a05b8fa9-f2cf-4809-8bea-47a84600c2b7" (UID: "a05b8fa9-f2cf-4809-8bea-47a84600c2b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:55.827683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.827661    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-socket" pod "a05b8fa9-f2cf-4809-8bea-47a84600c2b7" (UID: "a05b8fa9-f2cf-4809-8bea-47a84600c2b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:55.827756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.827686    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-credential-socket" pod "a05b8fa9-f2cf-4809-8bea-47a84600c2b7" (UID: "a05b8fa9-f2cf-4809-8bea-47a84600c2b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:55.827927 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.827902    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-certs" pod "a05b8fa9-f2cf-4809-8bea-47a84600c2b7" (UID: "a05b8fa9-f2cf-4809-8bea-47a84600c2b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:55.828295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.828268    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:55.828416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.828301    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:55.828416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.828318    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:55.828416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.828333    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:55.828416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.828349    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:55.829482 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.829458    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-podinfo" pod "a05b8fa9-f2cf-4809-8bea-47a84600c2b7" (UID: "a05b8fa9-f2cf-4809-8bea-47a84600c2b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:42:55.829767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.829644    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-envoy" pod "a05b8fa9-f2cf-4809-8bea-47a84600c2b7" (UID: "a05b8fa9-f2cf-4809-8bea-47a84600c2b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:42:55.829767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.829751    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-kube-api-access-6x7cx" pod "a05b8fa9-f2cf-4809-8bea-47a84600c2b7" (UID: "a05b8fa9-f2cf-4809-8bea-47a84600c2b7"). InnerVolumeSpecName "kube-api-access-6x7cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:42:55.829922 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.829782    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-token" pod "a05b8fa9-f2cf-4809-8bea-47a84600c2b7" (UID: "a05b8fa9-f2cf-4809-8bea-47a84600c2b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:42:55.929958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.929929    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:55.929958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.929955    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:55.930178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.929965    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x7cx\" (UniqueName: \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-kube-api-access-6x7cx\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:55.930178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:55.929976    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05b8fa9-f2cf-4809-8bea-47a84600c2b7-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:42:56.678290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:56.678258    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"
Nov 03 07:42:56.901122 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:56.901081    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"]
Nov 03 07:42:56.951836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:56.951764    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6"]
Nov 03 07:42:57.807244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:42:57.807214    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" path="/var/lib/kubelet/pods/a05b8fa9-f2cf-4809-8bea-47a84600c2b7/volumes"
Nov 03 07:43:04.760224 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:43:04.760183    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:43:14.456166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:14.456111    4470 scope.go:122] "RemoveContainer" containerID="901373d68f69089078a9d5964bca2f973991c6564ac38dde3c0a2161aef9fbd2"
Nov 03 07:43:14.466110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:14.466090    4470 scope.go:122] "RemoveContainer" containerID="13e55407d1b8fc3cc16416ad64065a60fa1c9f115112dae1d3f161a9281e5d21"
Nov 03 07:43:34.764804 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:43:34.764710    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:43:39.866584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:39.864690    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"]
Nov 03 07:43:39.911717 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:39.911687    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8983d-7vvdq"]
Nov 03 07:43:41.165115 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:41.165077    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"]
Nov 03 07:43:41.215211 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:41.215177    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fb4bc-5kcft"]
Nov 03 07:43:41.806420 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:41.806389    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14811088-b434-498a-8231-707393e1d590" path="/var/lib/kubelet/pods/14811088-b434-498a-8231-707393e1d590/volumes"
Nov 03 07:43:41.806863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:41.806851    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e4d65c-7d82-44bd-abe3-c60b05bd89fc" path="/var/lib/kubelet/pods/d0e4d65c-7d82-44bd-abe3-c60b05bd89fc/volumes"
Nov 03 07:43:49.377106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377067    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"]
Nov 03 07:43:49.377810 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377792    4470 cpu_manager.go:414] "containerMap: removing container" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy"
Nov 03 07:43:49.377859 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377815    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy"
Nov 03 07:43:49.377859 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377840    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy"
Nov 03 07:43:49.377859 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377848    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy"
Nov 03 07:43:49.377960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377867    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy"
Nov 03 07:43:49.377960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377872    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy"
Nov 03 07:43:49.377960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377891    4470 cpu_manager.go:414] "containerMap: removing container" podUID="14811088-b434-498a-8231-707393e1d590" containerName="revoke-keys"
Nov 03 07:43:49.377960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377895    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14811088-b434-498a-8231-707393e1d590" containerName="revoke-keys"
Nov 03 07:43:49.377960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377903    4470 cpu_manager.go:414] "containerMap: removing container" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy"
Nov 03 07:43:49.377960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377907    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy"
Nov 03 07:43:49.378200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.377996    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy"
Nov 03 07:43:49.378200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.378008    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="14811088-b434-498a-8231-707393e1d590" containerName="revoke-keys"
Nov 03 07:43:49.378200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.378015    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05b8fa9-f2cf-4809-8bea-47a84600c2b7" containerName="istio-proxy"
Nov 03 07:43:49.378200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.378023    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy"
Nov 03 07:43:49.378200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.378032    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy"
Nov 03 07:43:49.382294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.382277    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:49.407030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.407000    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:49.407241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.407091    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:49.407353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.407277    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkhkt\" (UniqueName: \"kubernetes.io/projected/8b134e69-9c3b-43c8-b418-2a845681536b-kube-api-access-qkhkt\") pod \"maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:49.440226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.440192    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"]
Nov 03 07:43:49.486194 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.486161    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:43:49.486387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.486324    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:43:49.486387 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.486343    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:43:49.487242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.487222    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:43:49.509049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.509022    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkhkt\" (UniqueName: \"kubernetes.io/projected/8b134e69-9c3b-43c8-b418-2a845681536b-kube-api-access-qkhkt\") pod \"maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:49.509261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.509104    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:49.509261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.509188    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:49.509793 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.509773    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:49.511846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.511818    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:49.576481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.576440    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkhkt\" (UniqueName: \"kubernetes.io/projected/8b134e69-9c3b-43c8-b418-2a845681536b-kube-api-access-qkhkt\") pod \"maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:49.695922 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:49.695839    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:50.083412 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.083379    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"]
Nov 03 07:43:50.084908 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:43:50.084879    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b134e69_9c3b_43c8_b418_2a845681536b.slice/crio-ca2563bd0efb7242cc57ed3c2302cd60896167622472eaa4cc5c1669c1521122 WatchSource:0}: Error finding container ca2563bd0efb7242cc57ed3c2302cd60896167622472eaa4cc5c1669c1521122: Status 404 returned error can't find the container with id ca2563bd0efb7242cc57ed3c2302cd60896167622472eaa4cc5c1669c1521122
Nov 03 07:43:50.493260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.493182    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"]
Nov 03 07:43:50.493744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.493729    4470 cpu_manager.go:414] "containerMap: removing container" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy"
Nov 03 07:43:50.493786 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.493745    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36405c56-94e9-4818-a713-ae9a8a677975" containerName="istio-proxy"
Nov 03 07:43:50.493858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.493849    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a557dbf-9da0-42bf-9354-38f46e3b9312" containerName="istio-proxy"
Nov 03 07:43:50.497341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.497326    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:50.530078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.530051    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:50.530266 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.530092    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mhf\" (UniqueName: \"kubernetes.io/projected/948812f6-4c5e-461b-b6a2-4d697f16c48f-kube-api-access-s9mhf\") pod \"maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:50.530339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.530315    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:50.558464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.558431    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"]
Nov 03 07:43:50.631908 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.631874    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:50.632090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.632055    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:50.632169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.632104    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mhf\" (UniqueName: \"kubernetes.io/projected/948812f6-4c5e-461b-b6a2-4d697f16c48f-kube-api-access-s9mhf\") pod \"maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:50.632741 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.632675    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:50.634468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.634445    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:50.696360 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.696328    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mhf\" (UniqueName: \"kubernetes.io/projected/948812f6-4c5e-461b-b6a2-4d697f16c48f-kube-api-access-s9mhf\") pod \"maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:50.811042 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:50.810964    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:51.073398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:51.073363    4470 generic.go:356] "Generic (PLEG): container finished" podID="8b134e69-9c3b-43c8-b418-2a845681536b" containerID="0dd6d06197f2fe83935575176b08459de13fbf325e70bb14dd31eaa28e516345" exitCode=0
Nov 03 07:43:51.073565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:51.073449    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t" event={"ID":"8b134e69-9c3b-43c8-b418-2a845681536b","Type":"ContainerDied","Data":"0dd6d06197f2fe83935575176b08459de13fbf325e70bb14dd31eaa28e516345"}
Nov 03 07:43:51.073565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:51.073503    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t" event={"ID":"8b134e69-9c3b-43c8-b418-2a845681536b","Type":"ContainerStarted","Data":"ca2563bd0efb7242cc57ed3c2302cd60896167622472eaa4cc5c1669c1521122"}
Nov 03 07:43:51.196048 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:51.196020    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"]
Nov 03 07:43:51.196983 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:43:51.196960    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948812f6_4c5e_461b_b6a2_4d697f16c48f.slice/crio-06e95abe549c74fc88b500e57f7ff8d07331798da91a041844e843af096e0465 WatchSource:0}: Error finding container 06e95abe549c74fc88b500e57f7ff8d07331798da91a041844e843af096e0465: Status 404 returned error can't find the container with id 06e95abe549c74fc88b500e57f7ff8d07331798da91a041844e843af096e0465
Nov 03 07:43:52.081500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.081462    4470 generic.go:356] "Generic (PLEG): container finished" podID="948812f6-4c5e-461b-b6a2-4d697f16c48f" containerID="6a2ee6982e5262fec9b210e5be5ca1922d888e71edae2e84bfef583f9d8a1f67" exitCode=0
Nov 03 07:43:52.081865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.081541    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb" event={"ID":"948812f6-4c5e-461b-b6a2-4d697f16c48f","Type":"ContainerDied","Data":"6a2ee6982e5262fec9b210e5be5ca1922d888e71edae2e84bfef583f9d8a1f67"}
Nov 03 07:43:52.081865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.081598    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb" event={"ID":"948812f6-4c5e-461b-b6a2-4d697f16c48f","Type":"ContainerStarted","Data":"06e95abe549c74fc88b500e57f7ff8d07331798da91a041844e843af096e0465"}
Nov 03 07:43:52.237204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.237178    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:52.367011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.366974    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-xks-ca\") pod \"8b134e69-9c3b-43c8-b418-2a845681536b\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") "
Nov 03 07:43:52.367245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.367087    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b134e69-9c3b-43c8-b418-2a845681536b-kube-api-access-qkhkt\" (UniqueName: \"kubernetes.io/projected/8b134e69-9c3b-43c8-b418-2a845681536b-kube-api-access-qkhkt\") pod \"8b134e69-9c3b-43c8-b418-2a845681536b\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") "
Nov 03 07:43:52.367245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.367234    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-openshift-ca\") pod \"8b134e69-9c3b-43c8-b418-2a845681536b\" (UID: \"8b134e69-9c3b-43c8-b418-2a845681536b\") "
Nov 03 07:43:52.367846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.367819    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-openshift-ca" pod "8b134e69-9c3b-43c8-b418-2a845681536b" (UID: "8b134e69-9c3b-43c8-b418-2a845681536b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:43:52.370045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.370014    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-xks-ca" pod "8b134e69-9c3b-43c8-b418-2a845681536b" (UID: "8b134e69-9c3b-43c8-b418-2a845681536b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:43:52.370194 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.370064    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b134e69-9c3b-43c8-b418-2a845681536b-kube-api-access-qkhkt" pod "8b134e69-9c3b-43c8-b418-2a845681536b" (UID: "8b134e69-9c3b-43c8-b418-2a845681536b"). InnerVolumeSpecName "kube-api-access-qkhkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:43:52.469164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.469101    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:43:52.469164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.469160    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkhkt\" (UniqueName: \"kubernetes.io/projected/8b134e69-9c3b-43c8-b418-2a845681536b-kube-api-access-qkhkt\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:43:52.469164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:52.469173    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b134e69-9c3b-43c8-b418-2a845681536b-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:43:53.089799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.089770    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"
Nov 03 07:43:53.090231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.089820    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t" event={"ID":"8b134e69-9c3b-43c8-b418-2a845681536b","Type":"ContainerDied","Data":"ca2563bd0efb7242cc57ed3c2302cd60896167622472eaa4cc5c1669c1521122"}
Nov 03 07:43:53.090231 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.089864    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca2563bd0efb7242cc57ed3c2302cd60896167622472eaa4cc5c1669c1521122"
Nov 03 07:43:53.247169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.247124    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:53.379732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.379663    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-openshift-ca\") pod \"948812f6-4c5e-461b-b6a2-4d697f16c48f\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") "
Nov 03 07:43:53.379880 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.379760    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-xks-ca\") pod \"948812f6-4c5e-461b-b6a2-4d697f16c48f\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") "
Nov 03 07:43:53.379931 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.379893    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948812f6-4c5e-461b-b6a2-4d697f16c48f-kube-api-access-s9mhf\" (UniqueName: \"kubernetes.io/projected/948812f6-4c5e-461b-b6a2-4d697f16c48f-kube-api-access-s9mhf\") pod \"948812f6-4c5e-461b-b6a2-4d697f16c48f\" (UID: \"948812f6-4c5e-461b-b6a2-4d697f16c48f\") "
Nov 03 07:43:53.380120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.380094    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-openshift-ca" pod "948812f6-4c5e-461b-b6a2-4d697f16c48f" (UID: "948812f6-4c5e-461b-b6a2-4d697f16c48f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:43:53.380357 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.380339    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:43:53.381994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.381972    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-xks-ca" pod "948812f6-4c5e-461b-b6a2-4d697f16c48f" (UID: "948812f6-4c5e-461b-b6a2-4d697f16c48f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:43:53.382615 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.382584    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948812f6-4c5e-461b-b6a2-4d697f16c48f-kube-api-access-s9mhf" pod "948812f6-4c5e-461b-b6a2-4d697f16c48f" (UID: "948812f6-4c5e-461b-b6a2-4d697f16c48f"). InnerVolumeSpecName "kube-api-access-s9mhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:43:53.481047 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.481015    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948812f6-4c5e-461b-b6a2-4d697f16c48f-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:43:53.481047 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:53.481045    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9mhf\" (UniqueName: \"kubernetes.io/projected/948812f6-4c5e-461b-b6a2-4d697f16c48f-kube-api-access-s9mhf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:43:54.098260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:54.098185    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"
Nov 03 07:43:54.098649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:54.098189    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb" event={"ID":"948812f6-4c5e-461b-b6a2-4d697f16c48f","Type":"ContainerDied","Data":"06e95abe549c74fc88b500e57f7ff8d07331798da91a041844e843af096e0465"}
Nov 03 07:43:54.098649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:43:54.098300    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06e95abe549c74fc88b500e57f7ff8d07331798da91a041844e843af096e0465"
Nov 03 07:44:00.919334 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:00.919299    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"]
Nov 03 07:44:01.027794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:01.027760    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"]
Nov 03 07:44:04.760292 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:44:04.760251    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:44:05.919859 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:05.919821    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h" podUID="c1eb7828-42fd-4c5a-80c8-a86cce97fd10" containerName="payload-processing" containerID="cri-o://d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741" gracePeriod=25
Nov 03 07:44:06.029312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.029266    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn" podUID="d5b6d9c1-75ff-455c-9b37-8df77fad98cb" containerName="payload-pre-processing" containerID="cri-o://888181339daa6cb190b2a595af176575815dd99d0180e0ea8922a693697f983b" gracePeriod=25
Nov 03 07:44:06.190920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.190856    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:44:06.194682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.194649    4470 generic.go:356] "Generic (PLEG): container finished" podID="c1eb7828-42fd-4c5a-80c8-a86cce97fd10" containerID="d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741" exitCode=0
Nov 03 07:44:06.194816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.194689    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"
Nov 03 07:44:06.194816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.194723    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h" event={"ID":"c1eb7828-42fd-4c5a-80c8-a86cce97fd10","Type":"ContainerDied","Data":"d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741"}
Nov 03 07:44:06.194816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.194783    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h" event={"ID":"c1eb7828-42fd-4c5a-80c8-a86cce97fd10","Type":"ContainerDied","Data":"a279b142ddba0328a648be709f4ccefca235a24a3482a31f26555431d63d084b"}
Nov 03 07:44:06.194816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.194807    4470 scope.go:122] "RemoveContainer" containerID="d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741"
Nov 03 07:44:06.196732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.196712    4470 generic.go:356] "Generic (PLEG): container finished" podID="d5b6d9c1-75ff-455c-9b37-8df77fad98cb" containerID="888181339daa6cb190b2a595af176575815dd99d0180e0ea8922a693697f983b" exitCode=0
Nov 03 07:44:06.196827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.196807    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn" event={"ID":"d5b6d9c1-75ff-455c-9b37-8df77fad98cb","Type":"ContainerDied","Data":"888181339daa6cb190b2a595af176575815dd99d0180e0ea8922a693697f983b"}
Nov 03 07:44:06.205407 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.205388    4470 scope.go:122] "RemoveContainer" containerID="d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741"
Nov 03 07:44:06.205757 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:44:06.205726    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741\": container with ID starting with d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741 not found: ID does not exist" containerID="d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741"
Nov 03 07:44:06.205875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.205764    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741"} err="failed to get container status \"d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741\": rpc error: code = NotFound desc = could not find container \"d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741\": container with ID starting with d3ea2e9945c63779b47ea4934f72bdc2f4ec670d371baab699f0d2f38c3eb741 not found: ID does not exist"
Nov 03 07:44:06.302104 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.302081    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:44:06.365573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.365538    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-kube-api-access-gv2mw\" (UniqueName: \"kubernetes.io/projected/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-kube-api-access-gv2mw\") pod \"c1eb7828-42fd-4c5a-80c8-a86cce97fd10\" (UID: \"c1eb7828-42fd-4c5a-80c8-a86cce97fd10\") "
Nov 03 07:44:06.365725 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.365650    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-plugins-config-volume\") pod \"c1eb7828-42fd-4c5a-80c8-a86cce97fd10\" (UID: \"c1eb7828-42fd-4c5a-80c8-a86cce97fd10\") "
Nov 03 07:44:06.366089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.366050    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-plugins-config-volume" pod "c1eb7828-42fd-4c5a-80c8-a86cce97fd10" (UID: "c1eb7828-42fd-4c5a-80c8-a86cce97fd10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:44:06.367756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.367732    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-kube-api-access-gv2mw" pod "c1eb7828-42fd-4c5a-80c8-a86cce97fd10" (UID: "c1eb7828-42fd-4c5a-80c8-a86cce97fd10"). InnerVolumeSpecName "kube-api-access-gv2mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:44:06.466844 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.466751    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-plugins-config-volume\") pod \"d5b6d9c1-75ff-455c-9b37-8df77fad98cb\" (UID: \"d5b6d9c1-75ff-455c-9b37-8df77fad98cb\") "
Nov 03 07:44:06.467013 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.466869    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-kube-api-access-mhj92\" (UniqueName: \"kubernetes.io/projected/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-kube-api-access-mhj92\") pod \"d5b6d9c1-75ff-455c-9b37-8df77fad98cb\" (UID: \"d5b6d9c1-75ff-455c-9b37-8df77fad98cb\") "
Nov 03 07:44:06.467223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.467197    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-plugins-config-volume" pod "d5b6d9c1-75ff-455c-9b37-8df77fad98cb" (UID: "d5b6d9c1-75ff-455c-9b37-8df77fad98cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:44:06.467473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.467452    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:44:06.467564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.467481    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv2mw\" (UniqueName: \"kubernetes.io/projected/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-kube-api-access-gv2mw\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:44:06.467564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.467497    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1eb7828-42fd-4c5a-80c8-a86cce97fd10-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:44:06.469363 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.469335    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-kube-api-access-mhj92" pod "d5b6d9c1-75ff-455c-9b37-8df77fad98cb" (UID: "d5b6d9c1-75ff-455c-9b37-8df77fad98cb"). InnerVolumeSpecName "kube-api-access-mhj92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:44:06.569236 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.569198    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhj92\" (UniqueName: \"kubernetes.io/projected/d5b6d9c1-75ff-455c-9b37-8df77fad98cb-kube-api-access-mhj92\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:44:06.623434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.623396    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"]
Nov 03 07:44:06.626385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:06.626360    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h"]
Nov 03 07:44:07.206472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:07.206391    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"
Nov 03 07:44:07.206472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:07.206408    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn" event={"ID":"d5b6d9c1-75ff-455c-9b37-8df77fad98cb","Type":"ContainerDied","Data":"349b7541c82235eabda972e23ea7583d9faee6f38df4f65ea741ed2cc8223bee"}
Nov 03 07:44:07.206939 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:07.206496    4470 scope.go:122] "RemoveContainer" containerID="888181339daa6cb190b2a595af176575815dd99d0180e0ea8922a693697f983b"
Nov 03 07:44:07.380002 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:07.379970    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"]
Nov 03 07:44:07.385037 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:07.385005    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn"]
Nov 03 07:44:07.807783 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:07.807752    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1eb7828-42fd-4c5a-80c8-a86cce97fd10" path="/var/lib/kubelet/pods/c1eb7828-42fd-4c5a-80c8-a86cce97fd10/volumes"
Nov 03 07:44:07.808218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:07.808204    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b6d9c1-75ff-455c-9b37-8df77fad98cb" path="/var/lib/kubelet/pods/d5b6d9c1-75ff-455c-9b37-8df77fad98cb/volumes"
Nov 03 07:44:11.057399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:11.057365    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"]
Nov 03 07:44:11.107928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:11.107890    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"]
Nov 03 07:44:11.110768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:11.110747    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-868fc96d-5gjv5"]
Nov 03 07:44:11.113227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:11.113207    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d93b019-2w86j"]
Nov 03 07:44:11.806533 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:11.806484    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b90089-4287-45be-a343-b3ed069e0188" path="/var/lib/kubelet/pods/40b90089-4287-45be-a343-b3ed069e0188/volumes"
Nov 03 07:44:11.807080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:11.807042    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad571469-adaa-4e64-b88a-6adeacfb6bd9" path="/var/lib/kubelet/pods/ad571469-adaa-4e64-b88a-6adeacfb6bd9/volumes"
Nov 03 07:44:14.508346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:14.508316    4470 scope.go:122] "RemoveContainer" containerID="32d78cd0cf634b947dc6c40b51f0de7268a3a6c4523db753883a85d851ad676f"
Nov 03 07:44:14.517848 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:44:14.517826    4470 scope.go:122] "RemoveContainer" containerID="e7644fe603f8ab81e21fea246cac15a6f13a9ad070704f51b08fd7d80d1aba3f"
Nov 03 07:44:34.759540 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:44:34.759507    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:45:00.191595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.191497    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"]
Nov 03 07:45:00.192358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192335    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8b134e69-9c3b-43c8-b418-2a845681536b" containerName="revoke-keys"
Nov 03 07:45:00.192485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192358    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b134e69-9c3b-43c8-b418-2a845681536b" containerName="revoke-keys"
Nov 03 07:45:00.192485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192373    4470 cpu_manager.go:414] "containerMap: removing container" podUID="948812f6-4c5e-461b-b6a2-4d697f16c48f" containerName="revoke-keys"
Nov 03 07:45:00.192485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192379    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948812f6-4c5e-461b-b6a2-4d697f16c48f" containerName="revoke-keys"
Nov 03 07:45:00.192485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192395    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c1eb7828-42fd-4c5a-80c8-a86cce97fd10" containerName="payload-processing"
Nov 03 07:45:00.192485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192403    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1eb7828-42fd-4c5a-80c8-a86cce97fd10" containerName="payload-processing"
Nov 03 07:45:00.192485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192430    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d5b6d9c1-75ff-455c-9b37-8df77fad98cb" containerName="payload-pre-processing"
Nov 03 07:45:00.192485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192438    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b6d9c1-75ff-455c-9b37-8df77fad98cb" containerName="payload-pre-processing"
Nov 03 07:45:00.192840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192592    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b134e69-9c3b-43c8-b418-2a845681536b" containerName="revoke-keys"
Nov 03 07:45:00.192840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192608    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="948812f6-4c5e-461b-b6a2-4d697f16c48f" containerName="revoke-keys"
Nov 03 07:45:00.192840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192619    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1eb7828-42fd-4c5a-80c8-a86cce97fd10" containerName="payload-processing"
Nov 03 07:45:00.192840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.192630    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b6d9c1-75ff-455c-9b37-8df77fad98cb" containerName="payload-pre-processing"
Nov 03 07:45:00.196192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.196170    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"]
Nov 03 07:45:00.196317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.196280    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"
Nov 03 07:45:00.200312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.200291    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"]
Nov 03 07:45:00.200486 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.200423    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"
Nov 03 07:45:00.204316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.204294    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"
Nov 03 07:45:00.242055 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.242024    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"]
Nov 03 07:45:00.246170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.246152    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"]
Nov 03 07:45:00.246309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.246273    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"
Nov 03 07:45:00.250100 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.250080    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"]
Nov 03 07:45:00.250239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.250221    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"
Nov 03 07:45:00.254526 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.254335    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"
Nov 03 07:45:00.255754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.255735    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"]
Nov 03 07:45:00.259381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.259365    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"
Nov 03 07:45:00.289702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.289664    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:45:00.289702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.289689    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:45:00.289913 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.289788    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:45:00.293439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.293416    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"]
Nov 03 07:45:00.296373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.296350    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"]
Nov 03 07:45:00.300464 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.300440    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"]
Nov 03 07:45:00.332474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.332434    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698ba69f-9dc7-4a28-ab83-8791b51d938f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2\" (UID: \"698ba69f-9dc7-4a28-ab83-8791b51d938f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"
Nov 03 07:45:00.332666 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.332556    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgqzd\" (UniqueName: \"kubernetes.io/projected/698ba69f-9dc7-4a28-ab83-8791b51d938f-kube-api-access-lgqzd\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2\" (UID: \"698ba69f-9dc7-4a28-ab83-8791b51d938f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"
Nov 03 07:45:00.332666 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.332650    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxpz6\" (UniqueName: \"kubernetes.io/projected/310e9a49-509b-4874-ac5e-dca0f8934fa2-kube-api-access-lxpz6\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7\" (UID: \"310e9a49-509b-4874-ac5e-dca0f8934fa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"
Nov 03 07:45:00.332826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.332683    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/310e9a49-509b-4874-ac5e-dca0f8934fa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7\" (UID: \"310e9a49-509b-4874-ac5e-dca0f8934fa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"
Nov 03 07:45:00.332826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.332730    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bf1202-fb69-421b-a075-7a5fdcec5f71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp\" (UID: \"76bf1202-fb69-421b-a075-7a5fdcec5f71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"
Nov 03 07:45:00.332826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.332781    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nv6p\" (UniqueName: \"kubernetes.io/projected/76bf1202-fb69-421b-a075-7a5fdcec5f71-kube-api-access-4nv6p\") pod \"maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp\" (UID: \"76bf1202-fb69-421b-a075-7a5fdcec5f71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"
Nov 03 07:45:00.349034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.348907    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"]
Nov 03 07:45:00.354668 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.354634    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"]
Nov 03 07:45:00.398289 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.398257    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"]
Nov 03 07:45:00.419622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.419578    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"]
Nov 03 07:45:00.434894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.434848    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvcwt\" (UniqueName: \"kubernetes.io/projected/f39be689-c2f3-4091-9097-5729f65ae5c0-kube-api-access-gvcwt\") pod \"maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2\" (UID: \"f39be689-c2f3-4091-9097-5729f65ae5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"
Nov 03 07:45:00.435081 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.434903    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfqcq\" (UniqueName: \"kubernetes.io/projected/cd4f0619-8935-4e73-83f9-5247d74f1897-kube-api-access-lfqcq\") pod \"maas-api-key-cleanup-29894865-p8jjx\" (UID: \"cd4f0619-8935-4e73-83f9-5247d74f1897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"
Nov 03 07:45:00.435081 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435041    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698ba69f-9dc7-4a28-ab83-8791b51d938f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2\" (UID: \"698ba69f-9dc7-4a28-ab83-8791b51d938f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"
Nov 03 07:45:00.435225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435203    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgqzd\" (UniqueName: \"kubernetes.io/projected/698ba69f-9dc7-4a28-ab83-8791b51d938f-kube-api-access-lgqzd\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2\" (UID: \"698ba69f-9dc7-4a28-ab83-8791b51d938f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"
Nov 03 07:45:00.435288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435267    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd4f0619-8935-4e73-83f9-5247d74f1897-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894865-p8jjx\" (UID: \"cd4f0619-8935-4e73-83f9-5247d74f1897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"
Nov 03 07:45:00.435347 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435314    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93bbaf9-6594-4956-bc86-854e3c012f78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz\" (UID: \"d93bbaf9-6594-4956-bc86-854e3c012f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"
Nov 03 07:45:00.435435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435409    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpz6\" (UniqueName: \"kubernetes.io/projected/310e9a49-509b-4874-ac5e-dca0f8934fa2-kube-api-access-lxpz6\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7\" (UID: \"310e9a49-509b-4874-ac5e-dca0f8934fa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"
Nov 03 07:45:00.435500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435455    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/310e9a49-509b-4874-ac5e-dca0f8934fa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7\" (UID: \"310e9a49-509b-4874-ac5e-dca0f8934fa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"
Nov 03 07:45:00.435500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435492    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39be689-c2f3-4091-9097-5729f65ae5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2\" (UID: \"f39be689-c2f3-4091-9097-5729f65ae5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"
Nov 03 07:45:00.435635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435534    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bf1202-fb69-421b-a075-7a5fdcec5f71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp\" (UID: \"76bf1202-fb69-421b-a075-7a5fdcec5f71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"
Nov 03 07:45:00.435635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435573    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb95462-5aed-4ee4-9223-06e45427a534-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj\" (UID: \"7fb95462-5aed-4ee4-9223-06e45427a534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"
Nov 03 07:45:00.435635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435617    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbmmt\" (UniqueName: \"kubernetes.io/projected/7fb95462-5aed-4ee4-9223-06e45427a534-kube-api-access-qbmmt\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj\" (UID: \"7fb95462-5aed-4ee4-9223-06e45427a534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"
Nov 03 07:45:00.435815 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435646    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nv6p\" (UniqueName: \"kubernetes.io/projected/76bf1202-fb69-421b-a075-7a5fdcec5f71-kube-api-access-4nv6p\") pod \"maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp\" (UID: \"76bf1202-fb69-421b-a075-7a5fdcec5f71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"
Nov 03 07:45:00.435815 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435676    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdl5\" (UniqueName: \"kubernetes.io/projected/d93bbaf9-6594-4956-bc86-854e3c012f78-kube-api-access-9xdl5\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz\" (UID: \"d93bbaf9-6594-4956-bc86-854e3c012f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"
Nov 03 07:45:00.435941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.435917    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698ba69f-9dc7-4a28-ab83-8791b51d938f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2\" (UID: \"698ba69f-9dc7-4a28-ab83-8791b51d938f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"
Nov 03 07:45:00.436307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.436284    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/310e9a49-509b-4874-ac5e-dca0f8934fa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7\" (UID: \"310e9a49-509b-4874-ac5e-dca0f8934fa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"
Nov 03 07:45:00.436524 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.436502    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bf1202-fb69-421b-a075-7a5fdcec5f71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp\" (UID: \"76bf1202-fb69-421b-a075-7a5fdcec5f71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"
Nov 03 07:45:00.505509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.505434    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxpz6\" (UniqueName: \"kubernetes.io/projected/310e9a49-509b-4874-ac5e-dca0f8934fa2-kube-api-access-lxpz6\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7\" (UID: \"310e9a49-509b-4874-ac5e-dca0f8934fa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"
Nov 03 07:45:00.506550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.506525    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nv6p\" (UniqueName: \"kubernetes.io/projected/76bf1202-fb69-421b-a075-7a5fdcec5f71-kube-api-access-4nv6p\") pod \"maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp\" (UID: \"76bf1202-fb69-421b-a075-7a5fdcec5f71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"
Nov 03 07:45:00.506657 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.506637    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgqzd\" (UniqueName: \"kubernetes.io/projected/698ba69f-9dc7-4a28-ab83-8791b51d938f-kube-api-access-lgqzd\") pod \"maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2\" (UID: \"698ba69f-9dc7-4a28-ab83-8791b51d938f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"
Nov 03 07:45:00.519552 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.519528    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"
Nov 03 07:45:00.527816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.527797    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"
Nov 03 07:45:00.535869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.535846    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"
Nov 03 07:45:00.537385 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.537363    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93bbaf9-6594-4956-bc86-854e3c012f78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz\" (UID: \"d93bbaf9-6594-4956-bc86-854e3c012f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"
Nov 03 07:45:00.537477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.537412    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39be689-c2f3-4091-9097-5729f65ae5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2\" (UID: \"f39be689-c2f3-4091-9097-5729f65ae5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"
Nov 03 07:45:00.537625 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.537606    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb95462-5aed-4ee4-9223-06e45427a534-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj\" (UID: \"7fb95462-5aed-4ee4-9223-06e45427a534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"
Nov 03 07:45:00.537722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.537661    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmmt\" (UniqueName: \"kubernetes.io/projected/7fb95462-5aed-4ee4-9223-06e45427a534-kube-api-access-qbmmt\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj\" (UID: \"7fb95462-5aed-4ee4-9223-06e45427a534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"
Nov 03 07:45:00.537722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.537700    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdl5\" (UniqueName: \"kubernetes.io/projected/d93bbaf9-6594-4956-bc86-854e3c012f78-kube-api-access-9xdl5\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz\" (UID: \"d93bbaf9-6594-4956-bc86-854e3c012f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"
Nov 03 07:45:00.537835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.537761    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcwt\" (UniqueName: \"kubernetes.io/projected/f39be689-c2f3-4091-9097-5729f65ae5c0-kube-api-access-gvcwt\") pod \"maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2\" (UID: \"f39be689-c2f3-4091-9097-5729f65ae5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"
Nov 03 07:45:00.537835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.537787    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfqcq\" (UniqueName: \"kubernetes.io/projected/cd4f0619-8935-4e73-83f9-5247d74f1897-kube-api-access-lfqcq\") pod \"maas-api-key-cleanup-29894865-p8jjx\" (UID: \"cd4f0619-8935-4e73-83f9-5247d74f1897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"
Nov 03 07:45:00.537975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.537961    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd4f0619-8935-4e73-83f9-5247d74f1897-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894865-p8jjx\" (UID: \"cd4f0619-8935-4e73-83f9-5247d74f1897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"
Nov 03 07:45:00.538120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.538086    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39be689-c2f3-4091-9097-5729f65ae5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2\" (UID: \"f39be689-c2f3-4091-9097-5729f65ae5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"
Nov 03 07:45:00.538376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.538350    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb95462-5aed-4ee4-9223-06e45427a534-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj\" (UID: \"7fb95462-5aed-4ee4-9223-06e45427a534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"
Nov 03 07:45:00.538736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.538712    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd4f0619-8935-4e73-83f9-5247d74f1897-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894865-p8jjx\" (UID: \"cd4f0619-8935-4e73-83f9-5247d74f1897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"
Nov 03 07:45:00.538895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.538873    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93bbaf9-6594-4956-bc86-854e3c012f78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz\" (UID: \"d93bbaf9-6594-4956-bc86-854e3c012f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"
Nov 03 07:45:00.614175 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.614037    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfqcq\" (UniqueName: \"kubernetes.io/projected/cd4f0619-8935-4e73-83f9-5247d74f1897-kube-api-access-lfqcq\") pod \"maas-api-key-cleanup-29894865-p8jjx\" (UID: \"cd4f0619-8935-4e73-83f9-5247d74f1897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"
Nov 03 07:45:00.614656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.614630    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdl5\" (UniqueName: \"kubernetes.io/projected/d93bbaf9-6594-4956-bc86-854e3c012f78-kube-api-access-9xdl5\") pod \"maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz\" (UID: \"d93bbaf9-6594-4956-bc86-854e3c012f78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"
Nov 03 07:45:00.615934 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.615905    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvcwt\" (UniqueName: \"kubernetes.io/projected/f39be689-c2f3-4091-9097-5729f65ae5c0-kube-api-access-gvcwt\") pod \"maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2\" (UID: \"f39be689-c2f3-4091-9097-5729f65ae5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"
Nov 03 07:45:00.616684 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.616398    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmmt\" (UniqueName: \"kubernetes.io/projected/7fb95462-5aed-4ee4-9223-06e45427a534-kube-api-access-qbmmt\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj\" (UID: \"7fb95462-5aed-4ee4-9223-06e45427a534\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"
Nov 03 07:45:00.862268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.862236    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"
Nov 03 07:45:00.869794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.869764    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"
Nov 03 07:45:00.878532 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.878505    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"
Nov 03 07:45:00.884714 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.884690    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"
Nov 03 07:45:00.915214 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:45:00.912767    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bf1202_fb69_421b_a075_7a5fdcec5f71.slice/crio-4c5eb10f7c5c2c75c3fee95a4923b5b812f0dbdde94c1ab3ddd2ee72a04d1954 WatchSource:0}: Error finding container 4c5eb10f7c5c2c75c3fee95a4923b5b812f0dbdde94c1ab3ddd2ee72a04d1954: Status 404 returned error can't find the container with id 4c5eb10f7c5c2c75c3fee95a4923b5b812f0dbdde94c1ab3ddd2ee72a04d1954
Nov 03 07:45:00.915214 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.913338    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"]
Nov 03 07:45:00.996222 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:00.996188    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"]
Nov 03 07:45:00.997542 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:45:00.996974    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698ba69f_9dc7_4a28_ab83_8791b51d938f.slice/crio-0c0183df4c142df5b608f1316bea6880dea51a05355a313ff839f63620c89534 WatchSource:0}: Error finding container 0c0183df4c142df5b608f1316bea6880dea51a05355a313ff839f63620c89534: Status 404 returned error can't find the container with id 0c0183df4c142df5b608f1316bea6880dea51a05355a313ff839f63620c89534
Nov 03 07:45:01.003111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.003087    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"]
Nov 03 07:45:01.265156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.265115    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"]
Nov 03 07:45:01.287660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.287634    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"]
Nov 03 07:45:01.305209 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.305186    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"]
Nov 03 07:45:01.315566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.315541    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"]
Nov 03 07:45:01.348915 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:45:01.348837    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb95462_5aed_4ee4_9223_06e45427a534.slice/crio-3be08bf39cf012899b6acc1a68669c3770a334b45e0fd282ab67c682a6750dc4 WatchSource:0}: Error finding container 3be08bf39cf012899b6acc1a68669c3770a334b45e0fd282ab67c682a6750dc4: Status 404 returned error can't find the container with id 3be08bf39cf012899b6acc1a68669c3770a334b45e0fd282ab67c682a6750dc4
Nov 03 07:45:01.351112 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:45:01.351085    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4f0619_8935_4e73_83f9_5247d74f1897.slice/crio-97a53d29e3cb6d4828d07fb018e8fe26c521ccd93361da253f1354b7e6c3b1dc WatchSource:0}: Error finding container 97a53d29e3cb6d4828d07fb018e8fe26c521ccd93361da253f1354b7e6c3b1dc: Status 404 returned error can't find the container with id 97a53d29e3cb6d4828d07fb018e8fe26c521ccd93361da253f1354b7e6c3b1dc
Nov 03 07:45:01.354874 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:45:01.354849    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39be689_c2f3_4091_9097_5729f65ae5c0.slice/crio-70ef94786b5060d528882fc2c8e6e9e30d33303a0973aa2286eed309b6d13d89 WatchSource:0}: Error finding container 70ef94786b5060d528882fc2c8e6e9e30d33303a0973aa2286eed309b6d13d89: Status 404 returned error can't find the container with id 70ef94786b5060d528882fc2c8e6e9e30d33303a0973aa2286eed309b6d13d89
Nov 03 07:45:01.582319 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.582286    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz" event={"ID":"d93bbaf9-6594-4956-bc86-854e3c012f78","Type":"ContainerStarted","Data":"e11b61638814d2fe6109696e597e0bcddcfa7608c8696f809c0316c55e83e3be"}
Nov 03 07:45:01.583972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.583859    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx" event={"ID":"cd4f0619-8935-4e73-83f9-5247d74f1897","Type":"ContainerStarted","Data":"97a53d29e3cb6d4828d07fb018e8fe26c521ccd93361da253f1354b7e6c3b1dc"}
Nov 03 07:45:01.585419 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.585393    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2" event={"ID":"f39be689-c2f3-4091-9097-5729f65ae5c0","Type":"ContainerStarted","Data":"70ef94786b5060d528882fc2c8e6e9e30d33303a0973aa2286eed309b6d13d89"}
Nov 03 07:45:01.586848 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.586823    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7" event={"ID":"310e9a49-509b-4874-ac5e-dca0f8934fa2","Type":"ContainerStarted","Data":"466b71b84192bb3a52a060af689724c1f0181e6e6bc1cab41072d09c5c05dd7d"}
Nov 03 07:45:01.586943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.586884    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7" event={"ID":"310e9a49-509b-4874-ac5e-dca0f8934fa2","Type":"ContainerStarted","Data":"193706051c165edb07e63e87f6345b3d59cf48e76c0cfcbed5267c5057363354"}
Nov 03 07:45:01.589223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.589195    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" event={"ID":"698ba69f-9dc7-4a28-ab83-8791b51d938f","Type":"ContainerStarted","Data":"ec42cc26c775bb912e9b2f7bb305ec189c4a1468200f07414318bf093d1bca93"}
Nov 03 07:45:01.589355 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.589245    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" event={"ID":"698ba69f-9dc7-4a28-ab83-8791b51d938f","Type":"ContainerStarted","Data":"0c0183df4c142df5b608f1316bea6880dea51a05355a313ff839f63620c89534"}
Nov 03 07:45:01.590853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.590832    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" event={"ID":"76bf1202-fb69-421b-a075-7a5fdcec5f71","Type":"ContainerStarted","Data":"6a9ad2fa7c3c6f96429e67ea70784d393a5bde4e39d2c0802b27910dec35fb57"}
Nov 03 07:45:01.590947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.590871    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" event={"ID":"76bf1202-fb69-421b-a075-7a5fdcec5f71","Type":"ContainerStarted","Data":"4c5eb10f7c5c2c75c3fee95a4923b5b812f0dbdde94c1ab3ddd2ee72a04d1954"}
Nov 03 07:45:01.592078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.592059    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj" event={"ID":"7fb95462-5aed-4ee4-9223-06e45427a534","Type":"ContainerStarted","Data":"3be08bf39cf012899b6acc1a68669c3770a334b45e0fd282ab67c682a6750dc4"}
Nov 03 07:45:01.707798 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.707697    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7" podStartSLOduration=1.707679387 podStartE2EDuration="1.707679387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:01.706104009 +0000 UTC m=+2754.038473006" watchObservedRunningTime="2026-11-03 07:45:01.707679387 +0000 UTC m=+2754.040048384"
Nov 03 07:45:01.826193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.826115    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" podStartSLOduration=1.8260995009999998 podStartE2EDuration="1.826099501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:01.825735671 +0000 UTC m=+2754.158104665" watchObservedRunningTime="2026-11-03 07:45:01.826099501 +0000 UTC m=+2754.158468494"
Nov 03 07:45:01.949183 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:01.947822    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" podStartSLOduration=1.947806719 podStartE2EDuration="1.947806719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:01.94527158 +0000 UTC m=+2754.277640584" watchObservedRunningTime="2026-11-03 07:45:01.947806719 +0000 UTC m=+2754.280175713"
Nov 03 07:45:02.601331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.601235    4470 generic.go:356] "Generic (PLEG): container finished" podID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerID="ec42cc26c775bb912e9b2f7bb305ec189c4a1468200f07414318bf093d1bca93" exitCode=60
Nov 03 07:45:02.601775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.601321    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" event={"ID":"698ba69f-9dc7-4a28-ab83-8791b51d938f","Type":"ContainerDied","Data":"ec42cc26c775bb912e9b2f7bb305ec189c4a1468200f07414318bf093d1bca93"}
Nov 03 07:45:02.601775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.601586    4470 scope.go:122] "RemoveContainer" containerID="ec42cc26c775bb912e9b2f7bb305ec189c4a1468200f07414318bf093d1bca93"
Nov 03 07:45:02.603022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.602999    4470 generic.go:356] "Generic (PLEG): container finished" podID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerID="6a9ad2fa7c3c6f96429e67ea70784d393a5bde4e39d2c0802b27910dec35fb57" exitCode=60
Nov 03 07:45:02.603145 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.603083    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" event={"ID":"76bf1202-fb69-421b-a075-7a5fdcec5f71","Type":"ContainerDied","Data":"6a9ad2fa7c3c6f96429e67ea70784d393a5bde4e39d2c0802b27910dec35fb57"}
Nov 03 07:45:02.603411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.603330    4470 scope.go:122] "RemoveContainer" containerID="6a9ad2fa7c3c6f96429e67ea70784d393a5bde4e39d2c0802b27910dec35fb57"
Nov 03 07:45:02.606247 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.606221    4470 generic.go:356] "Generic (PLEG): container finished" podID="7fb95462-5aed-4ee4-9223-06e45427a534" containerID="c7ab842a270ad828f5c3e22378bb233e2fa8612dce66a669dea3137066f8d3d2" exitCode=60
Nov 03 07:45:02.606372 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.606302    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj" event={"ID":"7fb95462-5aed-4ee4-9223-06e45427a534","Type":"ContainerDied","Data":"c7ab842a270ad828f5c3e22378bb233e2fa8612dce66a669dea3137066f8d3d2"}
Nov 03 07:45:02.606536 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.606522    4470 scope.go:122] "RemoveContainer" containerID="c7ab842a270ad828f5c3e22378bb233e2fa8612dce66a669dea3137066f8d3d2"
Nov 03 07:45:02.607975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.607952    4470 generic.go:356] "Generic (PLEG): container finished" podID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerID="a331cb3d341ba06103a699cd085a5d8d32cff080d1b598e955d93628106914fe" exitCode=60
Nov 03 07:45:02.608051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.608015    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz" event={"ID":"d93bbaf9-6594-4956-bc86-854e3c012f78","Type":"ContainerDied","Data":"a331cb3d341ba06103a699cd085a5d8d32cff080d1b598e955d93628106914fe"}
Nov 03 07:45:02.608425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.608323    4470 scope.go:122] "RemoveContainer" containerID="a331cb3d341ba06103a699cd085a5d8d32cff080d1b598e955d93628106914fe"
Nov 03 07:45:02.610172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.610123    4470 generic.go:356] "Generic (PLEG): container finished" podID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerID="86d9cf56b6a893c6431f9e13146c9f1edab11fea232376740b2ba4160d486d97" exitCode=60
Nov 03 07:45:02.610281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.610172    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx" event={"ID":"cd4f0619-8935-4e73-83f9-5247d74f1897","Type":"ContainerDied","Data":"86d9cf56b6a893c6431f9e13146c9f1edab11fea232376740b2ba4160d486d97"}
Nov 03 07:45:02.610515 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.610498    4470 scope.go:122] "RemoveContainer" containerID="86d9cf56b6a893c6431f9e13146c9f1edab11fea232376740b2ba4160d486d97"
Nov 03 07:45:02.612026 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.611789    4470 generic.go:356] "Generic (PLEG): container finished" podID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerID="70593b5748a18d6b36649b9d8ee1233e705cc0da7178493f28b0452f1aba9de2" exitCode=60
Nov 03 07:45:02.612026 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.611867    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2" event={"ID":"f39be689-c2f3-4091-9097-5729f65ae5c0","Type":"ContainerDied","Data":"70593b5748a18d6b36649b9d8ee1233e705cc0da7178493f28b0452f1aba9de2"}
Nov 03 07:45:02.612569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.612252    4470 scope.go:122] "RemoveContainer" containerID="70593b5748a18d6b36649b9d8ee1233e705cc0da7178493f28b0452f1aba9de2"
Nov 03 07:45:02.615913 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.615829    4470 generic.go:356] "Generic (PLEG): container finished" podID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerID="466b71b84192bb3a52a060af689724c1f0181e6e6bc1cab41072d09c5c05dd7d" exitCode=60
Nov 03 07:45:02.615963 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.615909    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7" event={"ID":"310e9a49-509b-4874-ac5e-dca0f8934fa2","Type":"ContainerDied","Data":"466b71b84192bb3a52a060af689724c1f0181e6e6bc1cab41072d09c5c05dd7d"}
Nov 03 07:45:02.616121 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:02.616109    4470 scope.go:122] "RemoveContainer" containerID="466b71b84192bb3a52a060af689724c1f0181e6e6bc1cab41072d09c5c05dd7d"
Nov 03 07:45:03.627711 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.627681    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz" event={"ID":"d93bbaf9-6594-4956-bc86-854e3c012f78","Type":"ContainerStarted","Data":"6f4571e84cb343be66cebc3be1c04f624d80b6013685d9e5b36c4fb0a74312ed"}
Nov 03 07:45:03.629706 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.629680    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx" event={"ID":"cd4f0619-8935-4e73-83f9-5247d74f1897","Type":"ContainerStarted","Data":"c95a10208d653a9e9d87113cf77c0861f45a0c30d64f6ec426b5a213f84e132b"}
Nov 03 07:45:03.631405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.631382    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2" event={"ID":"f39be689-c2f3-4091-9097-5729f65ae5c0","Type":"ContainerStarted","Data":"43c9d79f5616007cbb912b7a1500dff4a2ce424d27e3a368b4ddec41c29ee497"}
Nov 03 07:45:03.633163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.633117    4470 generic.go:356] "Generic (PLEG): container finished" podID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerID="c596cb105f12ca5bc6cb04cad27b2d8ad0f140e44bb9a8ae4232637ef8624860" exitCode=60
Nov 03 07:45:03.633163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.633156    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7" event={"ID":"310e9a49-509b-4874-ac5e-dca0f8934fa2","Type":"ContainerDied","Data":"c596cb105f12ca5bc6cb04cad27b2d8ad0f140e44bb9a8ae4232637ef8624860"}
Nov 03 07:45:03.633286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.633204    4470 scope.go:122] "RemoveContainer" containerID="466b71b84192bb3a52a060af689724c1f0181e6e6bc1cab41072d09c5c05dd7d"
Nov 03 07:45:03.633414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.633396    4470 scope.go:122] "RemoveContainer" containerID="c596cb105f12ca5bc6cb04cad27b2d8ad0f140e44bb9a8ae4232637ef8624860"
Nov 03 07:45:03.633651 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:03.633630    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7_odh-ai-gateway-infra(310e9a49-509b-4874-ac5e-dca0f8934fa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2"
Nov 03 07:45:03.635157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.635124    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" event={"ID":"698ba69f-9dc7-4a28-ab83-8791b51d938f","Type":"ContainerStarted","Data":"f445800a8e0d6f7a979d0e6705dea78a24b37785b25b531400505905b168e106"}
Nov 03 07:45:03.636948 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.636929    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" event={"ID":"76bf1202-fb69-421b-a075-7a5fdcec5f71","Type":"ContainerStarted","Data":"4861754688a15d331bb5b9b15a733f2d34e3674a153b06c6c841f299d08aaac6"}
Nov 03 07:45:03.638863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.638841    4470 generic.go:356] "Generic (PLEG): container finished" podID="7fb95462-5aed-4ee4-9223-06e45427a534" containerID="5017b27fc712eaccecbf043c9ba99e167a36f52a8c87e3fec09b218eb52546b2" exitCode=60
Nov 03 07:45:03.638967 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.638926    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj" event={"ID":"7fb95462-5aed-4ee4-9223-06e45427a534","Type":"ContainerDied","Data":"5017b27fc712eaccecbf043c9ba99e167a36f52a8c87e3fec09b218eb52546b2"}
Nov 03 07:45:03.639105 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.639092    4470 scope.go:122] "RemoveContainer" containerID="5017b27fc712eaccecbf043c9ba99e167a36f52a8c87e3fec09b218eb52546b2"
Nov 03 07:45:03.639333 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:03.639313    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj_odh-ai-gateway-infra(7fb95462-5aed-4ee4-9223-06e45427a534)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj" podUID="7fb95462-5aed-4ee4-9223-06e45427a534"
Nov 03 07:45:03.646263 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.646241    4470 scope.go:122] "RemoveContainer" containerID="c7ab842a270ad828f5c3e22378bb233e2fa8612dce66a669dea3137066f8d3d2"
Nov 03 07:45:03.749785 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.749732    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz" podStartSLOduration=3.749717726 podStartE2EDuration="3.749717726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:03.747109997 +0000 UTC m=+2756.079478988" watchObservedRunningTime="2026-11-03 07:45:03.749717726 +0000 UTC m=+2756.082086742"
Nov 03 07:45:03.866853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:03.866801    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2" podStartSLOduration=3.866786435 podStartE2EDuration="3.866786435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:03.865499336 +0000 UTC m=+2756.197868330" watchObservedRunningTime="2026-11-03 07:45:03.866786435 +0000 UTC m=+2756.199155471"
Nov 03 07:45:04.345897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.345842    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx" podStartSLOduration=4.345828221 podStartE2EDuration="4.345828221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:04.344501585 +0000 UTC m=+2756.676870581" watchObservedRunningTime="2026-11-03 07:45:04.345828221 +0000 UTC m=+2756.678197214"
Nov 03 07:45:04.373678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.373635    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"]
Nov 03 07:45:04.378128 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.373992    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr" podUID="59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" containerName="istio-proxy" containerID="cri-o://4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81" gracePeriod=30
Nov 03 07:45:04.648411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.648327    4470 generic.go:356] "Generic (PLEG): container finished" podID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerID="f445800a8e0d6f7a979d0e6705dea78a24b37785b25b531400505905b168e106" exitCode=60
Nov 03 07:45:04.648867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.648419    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" event={"ID":"698ba69f-9dc7-4a28-ab83-8791b51d938f","Type":"ContainerDied","Data":"f445800a8e0d6f7a979d0e6705dea78a24b37785b25b531400505905b168e106"}
Nov 03 07:45:04.648867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.648488    4470 scope.go:122] "RemoveContainer" containerID="ec42cc26c775bb912e9b2f7bb305ec189c4a1468200f07414318bf093d1bca93"
Nov 03 07:45:04.648867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.648720    4470 scope.go:122] "RemoveContainer" containerID="f445800a8e0d6f7a979d0e6705dea78a24b37785b25b531400505905b168e106"
Nov 03 07:45:04.649070 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:04.649050    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2_odh-ai-gateway-infra(698ba69f-9dc7-4a28-ab83-8791b51d938f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f"
Nov 03 07:45:04.650471 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.650448    4470 generic.go:356] "Generic (PLEG): container finished" podID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerID="4861754688a15d331bb5b9b15a733f2d34e3674a153b06c6c841f299d08aaac6" exitCode=60
Nov 03 07:45:04.650580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.650533    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" event={"ID":"76bf1202-fb69-421b-a075-7a5fdcec5f71","Type":"ContainerDied","Data":"4861754688a15d331bb5b9b15a733f2d34e3674a153b06c6c841f299d08aaac6"}
Nov 03 07:45:04.650767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.650752    4470 scope.go:122] "RemoveContainer" containerID="4861754688a15d331bb5b9b15a733f2d34e3674a153b06c6c841f299d08aaac6"
Nov 03 07:45:04.651056 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:04.651033    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp_odh-ai-gateway-infra(76bf1202-fb69-421b-a075-7a5fdcec5f71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71"
Nov 03 07:45:04.652325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.652309    4470 scope.go:122] "RemoveContainer" containerID="5017b27fc712eaccecbf043c9ba99e167a36f52a8c87e3fec09b218eb52546b2"
Nov 03 07:45:04.652549 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:04.652531    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj_odh-ai-gateway-infra(7fb95462-5aed-4ee4-9223-06e45427a534)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj" podUID="7fb95462-5aed-4ee4-9223-06e45427a534"
Nov 03 07:45:04.654049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.654028    4470 generic.go:356] "Generic (PLEG): container finished" podID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerID="6f4571e84cb343be66cebc3be1c04f624d80b6013685d9e5b36c4fb0a74312ed" exitCode=60
Nov 03 07:45:04.654164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.654110    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz" event={"ID":"d93bbaf9-6594-4956-bc86-854e3c012f78","Type":"ContainerDied","Data":"6f4571e84cb343be66cebc3be1c04f624d80b6013685d9e5b36c4fb0a74312ed"}
Nov 03 07:45:04.654351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.654336    4470 scope.go:122] "RemoveContainer" containerID="6f4571e84cb343be66cebc3be1c04f624d80b6013685d9e5b36c4fb0a74312ed"
Nov 03 07:45:04.654613 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:04.654590    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz_odh-ai-gateway-infra(d93bbaf9-6594-4956-bc86-854e3c012f78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78"
Nov 03 07:45:04.656026 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.656007    4470 generic.go:356] "Generic (PLEG): container finished" podID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerID="c95a10208d653a9e9d87113cf77c0861f45a0c30d64f6ec426b5a213f84e132b" exitCode=60
Nov 03 07:45:04.656125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.656030    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx" event={"ID":"cd4f0619-8935-4e73-83f9-5247d74f1897","Type":"ContainerDied","Data":"c95a10208d653a9e9d87113cf77c0861f45a0c30d64f6ec426b5a213f84e132b"}
Nov 03 07:45:04.656316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.656302    4470 scope.go:122] "RemoveContainer" containerID="c95a10208d653a9e9d87113cf77c0861f45a0c30d64f6ec426b5a213f84e132b"
Nov 03 07:45:04.656565 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:04.656548    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894865-p8jjx_odh-ai-gateway-infra(cd4f0619-8935-4e73-83f9-5247d74f1897)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897"
Nov 03 07:45:04.658051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.658027    4470 generic.go:356] "Generic (PLEG): container finished" podID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerID="43c9d79f5616007cbb912b7a1500dff4a2ce424d27e3a368b4ddec41c29ee497" exitCode=60
Nov 03 07:45:04.658179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.658100    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2" event={"ID":"f39be689-c2f3-4091-9097-5729f65ae5c0","Type":"ContainerDied","Data":"43c9d79f5616007cbb912b7a1500dff4a2ce424d27e3a368b4ddec41c29ee497"}
Nov 03 07:45:04.658443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.658425    4470 scope.go:122] "RemoveContainer" containerID="43c9d79f5616007cbb912b7a1500dff4a2ce424d27e3a368b4ddec41c29ee497"
Nov 03 07:45:04.658729 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:04.658707    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2_odh-ai-gateway-infra(f39be689-c2f3-4091-9097-5729f65ae5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0"
Nov 03 07:45:04.659966 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.659948    4470 scope.go:122] "RemoveContainer" containerID="c596cb105f12ca5bc6cb04cad27b2d8ad0f140e44bb9a8ae4232637ef8624860"
Nov 03 07:45:04.660290 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:04.660269    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7_odh-ai-gateway-infra(310e9a49-509b-4874-ac5e-dca0f8934fa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2"
Nov 03 07:45:04.662975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.662959    4470 scope.go:122] "RemoveContainer" containerID="6a9ad2fa7c3c6f96429e67ea70784d393a5bde4e39d2c0802b27910dec35fb57"
Nov 03 07:45:04.676416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.676397    4470 scope.go:122] "RemoveContainer" containerID="a331cb3d341ba06103a699cd085a5d8d32cff080d1b598e955d93628106914fe"
Nov 03 07:45:04.690083 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.690057    4470 scope.go:122] "RemoveContainer" containerID="86d9cf56b6a893c6431f9e13146c9f1edab11fea232376740b2ba4160d486d97"
Nov 03 07:45:04.703571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:04.703549    4470 scope.go:122] "RemoveContainer" containerID="70593b5748a18d6b36649b9d8ee1233e705cc0da7178493f28b0452f1aba9de2"
Nov 03 07:45:04.760201 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:04.760165    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:45:05.669957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:05.669929    4470 scope.go:122] "RemoveContainer" containerID="6f4571e84cb343be66cebc3be1c04f624d80b6013685d9e5b36c4fb0a74312ed"
Nov 03 07:45:05.670416 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:05.670293    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz_odh-ai-gateway-infra(d93bbaf9-6594-4956-bc86-854e3c012f78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78"
Nov 03 07:45:05.671592 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:05.671566    4470 scope.go:122] "RemoveContainer" containerID="c95a10208d653a9e9d87113cf77c0861f45a0c30d64f6ec426b5a213f84e132b"
Nov 03 07:45:05.671907 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:05.671885    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894865-p8jjx_odh-ai-gateway-infra(cd4f0619-8935-4e73-83f9-5247d74f1897)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897"
Nov 03 07:45:05.673119 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:05.673101    4470 scope.go:122] "RemoveContainer" containerID="43c9d79f5616007cbb912b7a1500dff4a2ce424d27e3a368b4ddec41c29ee497"
Nov 03 07:45:05.673354 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:05.673333    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2_odh-ai-gateway-infra(f39be689-c2f3-4091-9097-5729f65ae5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0"
Nov 03 07:45:05.674748 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:05.674730    4470 scope.go:122] "RemoveContainer" containerID="f445800a8e0d6f7a979d0e6705dea78a24b37785b25b531400505905b168e106"
Nov 03 07:45:05.674950 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:05.674927    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2_odh-ai-gateway-infra(698ba69f-9dc7-4a28-ab83-8791b51d938f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f"
Nov 03 07:45:05.676227 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:05.676210    4470 scope.go:122] "RemoveContainer" containerID="4861754688a15d331bb5b9b15a733f2d34e3674a153b06c6c841f299d08aaac6"
Nov 03 07:45:05.676411 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:05.676394    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp_odh-ai-gateway-infra(76bf1202-fb69-421b-a075-7a5fdcec5f71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71"
Nov 03 07:45:09.656033 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.655970    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:45:09.703599 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.703560    4470 generic.go:356] "Generic (PLEG): container finished" podID="59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" containerID="4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81" exitCode=0
Nov 03 07:45:09.703780 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.703619    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"
Nov 03 07:45:09.703780 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.703655    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr" event={"ID":"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df","Type":"ContainerDied","Data":"4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81"}
Nov 03 07:45:09.703780 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.703737    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr" event={"ID":"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df","Type":"ContainerDied","Data":"2df2f34b64f7e83f6531e80be5071236a963c8cc2cda4e2ac2511149a1af5a9a"}
Nov 03 07:45:09.703780 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.703752    4470 scope.go:122] "RemoveContainer" containerID="4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81"
Nov 03 07:45:09.713940 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.713923    4470 scope.go:122] "RemoveContainer" containerID="4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81"
Nov 03 07:45:09.714262 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:09.714236    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81\": container with ID starting with 4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81 not found: ID does not exist" containerID="4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81"
Nov 03 07:45:09.714314 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.714270    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81"} err="failed to get container status \"4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81\": rpc error: code = NotFound desc = could not find container \"4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81\": container with ID starting with 4ae012e9cd5b977bacbae2ecba0abc793dbd04a7a623087bedb34bdefbcd5d81 not found: ID does not exist"
Nov 03 07:45:09.771665 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.771629    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-credential-socket\") pod \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") "
Nov 03 07:45:09.771801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.771779    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-socket\") pod \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") "
Nov 03 07:45:09.771863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.771812    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-certs\") pod \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") "
Nov 03 07:45:09.771863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.771840    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-data\") pod \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") "
Nov 03 07:45:09.771980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.771867    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-podinfo\") pod \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") "
Nov 03 07:45:09.771980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.771890    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-credential-socket" pod "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" (UID: "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:09.771980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.771925    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-token\" (UniqueName: \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-token\") pod \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") "
Nov 03 07:45:09.771980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.771951    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istiod-ca-cert\") pod \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") "
Nov 03 07:45:09.772241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.771982    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-kube-api-access-m997f\" (UniqueName: \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-kube-api-access-m997f\") pod \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") "
Nov 03 07:45:09.772241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.772076    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-envoy\") pod \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\" (UID: \"59dbf6a0-8d6e-4cbc-bde3-7b665e9021df\") "
Nov 03 07:45:09.772241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.772181    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-data" pod "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" (UID: "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:09.772241 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.772229    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-certs" pod "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" (UID: "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:09.772504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.772443    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istiod-ca-cert" pod "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" (UID: "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:09.772575 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.772518    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-socket" pod "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" (UID: "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:09.772957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.772684    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:09.772957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.772712    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:09.772957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.772767    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:09.772957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.772861    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:09.772957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.772913    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:09.774483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.774443    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-kube-api-access-m997f" pod "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" (UID: "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df"). InnerVolumeSpecName "kube-api-access-m997f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:09.774707 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.774678    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-envoy" pod "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" (UID: "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:09.774840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.774822    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-podinfo" pod "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" (UID: "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:45:09.774900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.774878    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-token" pod "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" (UID: "59dbf6a0-8d6e-4cbc-bde3-7b665e9021df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:09.875212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.874705    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:09.875212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.874739    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:09.875212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.874755    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:09.875212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:09.874768    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m997f\" (UniqueName: \"kubernetes.io/projected/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df-kube-api-access-m997f\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:10.126698 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:10.126663    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"]
Nov 03 07:45:10.132740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:10.132709    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr"]
Nov 03 07:45:11.807812 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:11.807784    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" path="/var/lib/kubelet/pods/59dbf6a0-8d6e-4cbc-bde3-7b665e9021df/volumes"
Nov 03 07:45:14.563745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:14.563711    4470 scope.go:122] "RemoveContainer" containerID="942f047431d1eb500330617c9f02cd604ba6c4aa4b37670afa0ab6644ae43415"
Nov 03 07:45:14.573272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:14.573249    4470 scope.go:122] "RemoveContainer" containerID="4a3458df700483a3ee8a683fc1df81ebad80536dd839a52db7cb02b58565ceb3"
Nov 03 07:45:14.582094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:14.582074    4470 scope.go:122] "RemoveContainer" containerID="ecb64939aa90edd50a4172c47c5605144e1b762ec3eea3af5ea1a780fe2efc8a"
Nov 03 07:45:14.799615 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:14.799580    4470 scope.go:122] "RemoveContainer" containerID="5017b27fc712eaccecbf043c9ba99e167a36f52a8c87e3fec09b218eb52546b2"
Nov 03 07:45:14.799785 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:14.799721    4470 scope.go:122] "RemoveContainer" containerID="6f4571e84cb343be66cebc3be1c04f624d80b6013685d9e5b36c4fb0a74312ed"
Nov 03 07:45:14.799889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:14.799871    4470 scope.go:122] "RemoveContainer" containerID="f445800a8e0d6f7a979d0e6705dea78a24b37785b25b531400505905b168e106"
Nov 03 07:45:14.800026 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:14.800007    4470 scope.go:122] "RemoveContainer" containerID="43c9d79f5616007cbb912b7a1500dff4a2ce424d27e3a368b4ddec41c29ee497"
Nov 03 07:45:14.800225 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:14.800199    4470 scope.go:122] "RemoveContainer" containerID="4861754688a15d331bb5b9b15a733f2d34e3674a153b06c6c841f299d08aaac6"
Nov 03 07:45:15.745373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:15.745336    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz" event={"ID":"d93bbaf9-6594-4956-bc86-854e3c012f78","Type":"ContainerStarted","Data":"26162ed77ba097f7d1d281fb342694bb416819e94f66e0a1d3ca96f476252afb"}
Nov 03 07:45:15.750162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:15.750100    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2" event={"ID":"f39be689-c2f3-4091-9097-5729f65ae5c0","Type":"ContainerStarted","Data":"360519caf0576b5975edb8034fc487a6c83ea28602901c5ba711e324ef428b14"}
Nov 03 07:45:15.753570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:15.753547    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" event={"ID":"698ba69f-9dc7-4a28-ab83-8791b51d938f","Type":"ContainerStarted","Data":"388086c0b6d252546f5736a8200780efedeb0796eaf8b38c58efb6ffb07d4312"}
Nov 03 07:45:15.756753 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:15.756726    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" event={"ID":"76bf1202-fb69-421b-a075-7a5fdcec5f71","Type":"ContainerStarted","Data":"17159b6e6cbe9644806522c207ffc1fb20435a65b7c1a19e3dab98b20bca02d9"}
Nov 03 07:45:15.758767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:15.758746    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj" event={"ID":"7fb95462-5aed-4ee4-9223-06e45427a534","Type":"ContainerStarted","Data":"b1a8e6c21ca46709cba9697806e567099d0253e3ad33555ca021e73059100d7d"}
Nov 03 07:45:16.234167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.234098    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj" podStartSLOduration=16.234083789 podStartE2EDuration="16.234083789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:16.231219206 +0000 UTC m=+2768.563588197" watchObservedRunningTime="2026-11-03 07:45:16.234083789 +0000 UTC m=+2768.566452788"
Nov 03 07:45:16.766581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.766550    4470 generic.go:356] "Generic (PLEG): container finished" podID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerID="26162ed77ba097f7d1d281fb342694bb416819e94f66e0a1d3ca96f476252afb" exitCode=60
Nov 03 07:45:16.767079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.766625    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz" event={"ID":"d93bbaf9-6594-4956-bc86-854e3c012f78","Type":"ContainerDied","Data":"26162ed77ba097f7d1d281fb342694bb416819e94f66e0a1d3ca96f476252afb"}
Nov 03 07:45:16.767079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.766711    4470 scope.go:122] "RemoveContainer" containerID="6f4571e84cb343be66cebc3be1c04f624d80b6013685d9e5b36c4fb0a74312ed"
Nov 03 07:45:16.767218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.767119    4470 scope.go:122] "RemoveContainer" containerID="26162ed77ba097f7d1d281fb342694bb416819e94f66e0a1d3ca96f476252afb"
Nov 03 07:45:16.767506 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:16.767475    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz_odh-ai-gateway-infra(d93bbaf9-6594-4956-bc86-854e3c012f78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78"
Nov 03 07:45:16.768722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.768694    4470 generic.go:356] "Generic (PLEG): container finished" podID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerID="360519caf0576b5975edb8034fc487a6c83ea28602901c5ba711e324ef428b14" exitCode=60
Nov 03 07:45:16.768808 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.768755    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2" event={"ID":"f39be689-c2f3-4091-9097-5729f65ae5c0","Type":"ContainerDied","Data":"360519caf0576b5975edb8034fc487a6c83ea28602901c5ba711e324ef428b14"}
Nov 03 07:45:16.769176 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.769160    4470 scope.go:122] "RemoveContainer" containerID="360519caf0576b5975edb8034fc487a6c83ea28602901c5ba711e324ef428b14"
Nov 03 07:45:16.769472 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:16.769451    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2_odh-ai-gateway-infra(f39be689-c2f3-4091-9097-5729f65ae5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0"
Nov 03 07:45:16.770663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.770644    4470 generic.go:356] "Generic (PLEG): container finished" podID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerID="388086c0b6d252546f5736a8200780efedeb0796eaf8b38c58efb6ffb07d4312" exitCode=60
Nov 03 07:45:16.770759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.770698    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" event={"ID":"698ba69f-9dc7-4a28-ab83-8791b51d938f","Type":"ContainerDied","Data":"388086c0b6d252546f5736a8200780efedeb0796eaf8b38c58efb6ffb07d4312"}
Nov 03 07:45:16.770984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.770969    4470 scope.go:122] "RemoveContainer" containerID="388086c0b6d252546f5736a8200780efedeb0796eaf8b38c58efb6ffb07d4312"
Nov 03 07:45:16.771271 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:16.771251    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2_odh-ai-gateway-infra(698ba69f-9dc7-4a28-ab83-8791b51d938f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f"
Nov 03 07:45:16.772660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.772643    4470 generic.go:356] "Generic (PLEG): container finished" podID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerID="17159b6e6cbe9644806522c207ffc1fb20435a65b7c1a19e3dab98b20bca02d9" exitCode=60
Nov 03 07:45:16.772768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.772714    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" event={"ID":"76bf1202-fb69-421b-a075-7a5fdcec5f71","Type":"ContainerDied","Data":"17159b6e6cbe9644806522c207ffc1fb20435a65b7c1a19e3dab98b20bca02d9"}
Nov 03 07:45:16.772997 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.772967    4470 scope.go:122] "RemoveContainer" containerID="17159b6e6cbe9644806522c207ffc1fb20435a65b7c1a19e3dab98b20bca02d9"
Nov 03 07:45:16.773267 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:16.773246    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp_odh-ai-gateway-infra(76bf1202-fb69-421b-a075-7a5fdcec5f71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71"
Nov 03 07:45:16.774504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.774487    4470 generic.go:356] "Generic (PLEG): container finished" podID="7fb95462-5aed-4ee4-9223-06e45427a534" containerID="b1a8e6c21ca46709cba9697806e567099d0253e3ad33555ca021e73059100d7d" exitCode=60
Nov 03 07:45:16.774593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.774520    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj" event={"ID":"7fb95462-5aed-4ee4-9223-06e45427a534","Type":"ContainerDied","Data":"b1a8e6c21ca46709cba9697806e567099d0253e3ad33555ca021e73059100d7d"}
Nov 03 07:45:16.774752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.774740    4470 scope.go:122] "RemoveContainer" containerID="b1a8e6c21ca46709cba9697806e567099d0253e3ad33555ca021e73059100d7d"
Nov 03 07:45:16.774946 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:16.774925    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj_odh-ai-gateway-infra(7fb95462-5aed-4ee4-9223-06e45427a534)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj" podUID="7fb95462-5aed-4ee4-9223-06e45427a534"
Nov 03 07:45:16.780169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.780152    4470 scope.go:122] "RemoveContainer" containerID="43c9d79f5616007cbb912b7a1500dff4a2ce424d27e3a368b4ddec41c29ee497"
Nov 03 07:45:16.793364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.793347    4470 scope.go:122] "RemoveContainer" containerID="f445800a8e0d6f7a979d0e6705dea78a24b37785b25b531400505905b168e106"
Nov 03 07:45:16.799686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.799668    4470 scope.go:122] "RemoveContainer" containerID="c596cb105f12ca5bc6cb04cad27b2d8ad0f140e44bb9a8ae4232637ef8624860"
Nov 03 07:45:16.806284 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.806266    4470 scope.go:122] "RemoveContainer" containerID="4861754688a15d331bb5b9b15a733f2d34e3674a153b06c6c841f299d08aaac6"
Nov 03 07:45:16.831292 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.831275    4470 scope.go:122] "RemoveContainer" containerID="5017b27fc712eaccecbf043c9ba99e167a36f52a8c87e3fec09b218eb52546b2"
Nov 03 07:45:16.880694 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:16.880664    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"]
Nov 03 07:45:17.001558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.001528    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"]
Nov 03 07:45:17.096234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.096202    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"]
Nov 03 07:45:17.100816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.100791    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8da554f-kpg22"]
Nov 03 07:45:17.120234 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.120204    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"]
Nov 03 07:45:17.244900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.244869    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"]
Nov 03 07:45:17.373517 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.373473    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"]
Nov 03 07:45:17.797535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.797508    4470 generic.go:356] "Generic (PLEG): container finished" podID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerID="3c5666d8b1c028242c0a1053f062a038f5889410e57e35acfd5226a881990f87" exitCode=60
Nov 03 07:45:17.797952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.797562    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7" event={"ID":"310e9a49-509b-4874-ac5e-dca0f8934fa2","Type":"ContainerDied","Data":"3c5666d8b1c028242c0a1053f062a038f5889410e57e35acfd5226a881990f87"}
Nov 03 07:45:17.797952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.797610    4470 scope.go:122] "RemoveContainer" containerID="c596cb105f12ca5bc6cb04cad27b2d8ad0f140e44bb9a8ae4232637ef8624860"
Nov 03 07:45:17.798077 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.798021    4470 scope.go:122] "RemoveContainer" containerID="3c5666d8b1c028242c0a1053f062a038f5889410e57e35acfd5226a881990f87"
Nov 03 07:45:17.798381 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:17.798355    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7_odh-ai-gateway-infra(310e9a49-509b-4874-ac5e-dca0f8934fa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2"
Nov 03 07:45:17.799064 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.799046    4470 scope.go:122] "RemoveContainer" containerID="c95a10208d653a9e9d87113cf77c0861f45a0c30d64f6ec426b5a213f84e132b"
Nov 03 07:45:17.831011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.830003    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2d9cb9-8218-43ca-a05f-20c711e594f6" path="/var/lib/kubelet/pods/ef2d9cb9-8218-43ca-a05f-20c711e594f6/volumes"
Nov 03 07:45:17.975293 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:17.975268    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"
Nov 03 07:45:18.016739 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.016707    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"
Nov 03 07:45:18.064037 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.063420    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"]
Nov 03 07:45:18.070993 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.070967    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5374f9-hsb9w"]
Nov 03 07:45:18.072948 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.072924    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"
Nov 03 07:45:18.085478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.085453    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"
Nov 03 07:45:18.089506 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.089488    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"
Nov 03 07:45:18.096627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.096594    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698ba69f-9dc7-4a28-ab83-8791b51d938f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698ba69f-9dc7-4a28-ab83-8791b51d938f-maas-api-service-ca\") pod \"698ba69f-9dc7-4a28-ab83-8791b51d938f\" (UID: \"698ba69f-9dc7-4a28-ab83-8791b51d938f\") "
Nov 03 07:45:18.096779 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.096667    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698ba69f-9dc7-4a28-ab83-8791b51d938f-kube-api-access-lgqzd\" (UniqueName: \"kubernetes.io/projected/698ba69f-9dc7-4a28-ab83-8791b51d938f-kube-api-access-lgqzd\") pod \"698ba69f-9dc7-4a28-ab83-8791b51d938f\" (UID: \"698ba69f-9dc7-4a28-ab83-8791b51d938f\") "
Nov 03 07:45:18.096848 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.096836    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93bbaf9-6594-4956-bc86-854e3c012f78-kube-api-access-9xdl5\" (UniqueName: \"kubernetes.io/projected/d93bbaf9-6594-4956-bc86-854e3c012f78-kube-api-access-9xdl5\") pod \"d93bbaf9-6594-4956-bc86-854e3c012f78\" (UID: \"d93bbaf9-6594-4956-bc86-854e3c012f78\") "
Nov 03 07:45:18.096949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.096934    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d93bbaf9-6594-4956-bc86-854e3c012f78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93bbaf9-6594-4956-bc86-854e3c012f78-maas-api-service-ca\") pod \"d93bbaf9-6594-4956-bc86-854e3c012f78\" (UID: \"d93bbaf9-6594-4956-bc86-854e3c012f78\") "
Nov 03 07:45:18.097412 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.097366    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698ba69f-9dc7-4a28-ab83-8791b51d938f-maas-api-service-ca" pod "698ba69f-9dc7-4a28-ab83-8791b51d938f" (UID: "698ba69f-9dc7-4a28-ab83-8791b51d938f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:18.097615 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.097588    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93bbaf9-6594-4956-bc86-854e3c012f78-maas-api-service-ca" pod "d93bbaf9-6594-4956-bc86-854e3c012f78" (UID: "d93bbaf9-6594-4956-bc86-854e3c012f78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:18.099822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.099800    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698ba69f-9dc7-4a28-ab83-8791b51d938f-kube-api-access-lgqzd" pod "698ba69f-9dc7-4a28-ab83-8791b51d938f" (UID: "698ba69f-9dc7-4a28-ab83-8791b51d938f"). InnerVolumeSpecName "kube-api-access-lgqzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:18.099939 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.099823    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93bbaf9-6594-4956-bc86-854e3c012f78-kube-api-access-9xdl5" pod "d93bbaf9-6594-4956-bc86-854e3c012f78" (UID: "d93bbaf9-6594-4956-bc86-854e3c012f78"). InnerVolumeSpecName "kube-api-access-9xdl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:18.197846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.197800    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f39be689-c2f3-4091-9097-5729f65ae5c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39be689-c2f3-4091-9097-5729f65ae5c0-maas-api-service-ca\") pod \"f39be689-c2f3-4091-9097-5729f65ae5c0\" (UID: \"f39be689-c2f3-4091-9097-5729f65ae5c0\") "
Nov 03 07:45:18.198011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.197927    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb95462-5aed-4ee4-9223-06e45427a534-kube-api-access-qbmmt\" (UniqueName: \"kubernetes.io/projected/7fb95462-5aed-4ee4-9223-06e45427a534-kube-api-access-qbmmt\") pod \"7fb95462-5aed-4ee4-9223-06e45427a534\" (UID: \"7fb95462-5aed-4ee4-9223-06e45427a534\") "
Nov 03 07:45:18.198011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.197997    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bf1202-fb69-421b-a075-7a5fdcec5f71-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bf1202-fb69-421b-a075-7a5fdcec5f71-maas-api-service-ca\") pod \"76bf1202-fb69-421b-a075-7a5fdcec5f71\" (UID: \"76bf1202-fb69-421b-a075-7a5fdcec5f71\") "
Nov 03 07:45:18.198125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198059    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb95462-5aed-4ee4-9223-06e45427a534-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb95462-5aed-4ee4-9223-06e45427a534-maas-api-service-ca\") pod \"7fb95462-5aed-4ee4-9223-06e45427a534\" (UID: \"7fb95462-5aed-4ee4-9223-06e45427a534\") "
Nov 03 07:45:18.198125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198099    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f39be689-c2f3-4091-9097-5729f65ae5c0-kube-api-access-gvcwt\" (UniqueName: \"kubernetes.io/projected/f39be689-c2f3-4091-9097-5729f65ae5c0-kube-api-access-gvcwt\") pod \"f39be689-c2f3-4091-9097-5729f65ae5c0\" (UID: \"f39be689-c2f3-4091-9097-5729f65ae5c0\") "
Nov 03 07:45:18.198125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198120    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bf1202-fb69-421b-a075-7a5fdcec5f71-kube-api-access-4nv6p\" (UniqueName: \"kubernetes.io/projected/76bf1202-fb69-421b-a075-7a5fdcec5f71-kube-api-access-4nv6p\") pod \"76bf1202-fb69-421b-a075-7a5fdcec5f71\" (UID: \"76bf1202-fb69-421b-a075-7a5fdcec5f71\") "
Nov 03 07:45:18.198335 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198260    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39be689-c2f3-4091-9097-5729f65ae5c0-maas-api-service-ca" pod "f39be689-c2f3-4091-9097-5729f65ae5c0" (UID: "f39be689-c2f3-4091-9097-5729f65ae5c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:18.198439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198413    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bf1202-fb69-421b-a075-7a5fdcec5f71-maas-api-service-ca" pod "76bf1202-fb69-421b-a075-7a5fdcec5f71" (UID: "76bf1202-fb69-421b-a075-7a5fdcec5f71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:18.198506 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198473    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb95462-5aed-4ee4-9223-06e45427a534-maas-api-service-ca" pod "7fb95462-5aed-4ee4-9223-06e45427a534" (UID: "7fb95462-5aed-4ee4-9223-06e45427a534"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:18.198676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198651    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xdl5\" (UniqueName: \"kubernetes.io/projected/d93bbaf9-6594-4956-bc86-854e3c012f78-kube-api-access-9xdl5\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:18.198676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198676    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bf1202-fb69-421b-a075-7a5fdcec5f71-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:18.199032 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198690    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb95462-5aed-4ee4-9223-06e45427a534-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:18.199032 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198700    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93bbaf9-6594-4956-bc86-854e3c012f78-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:18.199032 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198708    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698ba69f-9dc7-4a28-ab83-8791b51d938f-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:18.199032 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198718    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgqzd\" (UniqueName: \"kubernetes.io/projected/698ba69f-9dc7-4a28-ab83-8791b51d938f-kube-api-access-lgqzd\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:18.199032 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.198726    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39be689-c2f3-4091-9097-5729f65ae5c0-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:18.200540 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.200518    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39be689-c2f3-4091-9097-5729f65ae5c0-kube-api-access-gvcwt" pod "f39be689-c2f3-4091-9097-5729f65ae5c0" (UID: "f39be689-c2f3-4091-9097-5729f65ae5c0"). InnerVolumeSpecName "kube-api-access-gvcwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:18.200637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.200559    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb95462-5aed-4ee4-9223-06e45427a534-kube-api-access-qbmmt" pod "7fb95462-5aed-4ee4-9223-06e45427a534" (UID: "7fb95462-5aed-4ee4-9223-06e45427a534"). InnerVolumeSpecName "kube-api-access-qbmmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:18.200637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.200575    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bf1202-fb69-421b-a075-7a5fdcec5f71-kube-api-access-4nv6p" pod "76bf1202-fb69-421b-a075-7a5fdcec5f71" (UID: "76bf1202-fb69-421b-a075-7a5fdcec5f71"). InnerVolumeSpecName "kube-api-access-4nv6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:18.299688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.299649    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbmmt\" (UniqueName: \"kubernetes.io/projected/7fb95462-5aed-4ee4-9223-06e45427a534-kube-api-access-qbmmt\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:18.299688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.299684    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvcwt\" (UniqueName: \"kubernetes.io/projected/f39be689-c2f3-4091-9097-5729f65ae5c0-kube-api-access-gvcwt\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:18.299889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.299699    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nv6p\" (UniqueName: \"kubernetes.io/projected/76bf1202-fb69-421b-a075-7a5fdcec5f71-kube-api-access-4nv6p\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:18.496847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.496812    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"]
Nov 03 07:45:18.497013 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.496993    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc" podUID="9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" containerName="discovery" containerID="cri-o://a0b905d331b6f0ac173c904c8a6d34dbcc52281b09e30a5c42d4437c28883031" gracePeriod=30
Nov 03 07:45:18.819868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.819291    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"
Nov 03 07:45:18.820670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.820606    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2" event={"ID":"698ba69f-9dc7-4a28-ab83-8791b51d938f","Type":"ContainerDied","Data":"0c0183df4c142df5b608f1316bea6880dea51a05355a313ff839f63620c89534"}
Nov 03 07:45:18.820746 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.820706    4470 scope.go:122] "RemoveContainer" containerID="388086c0b6d252546f5736a8200780efedeb0796eaf8b38c58efb6ffb07d4312"
Nov 03 07:45:18.825593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.825556    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp" event={"ID":"76bf1202-fb69-421b-a075-7a5fdcec5f71","Type":"ContainerDied","Data":"4c5eb10f7c5c2c75c3fee95a4923b5b812f0dbdde94c1ab3ddd2ee72a04d1954"}
Nov 03 07:45:18.825721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.825567    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"
Nov 03 07:45:18.827830 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.827395    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"
Nov 03 07:45:18.827830 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.827420    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj" event={"ID":"7fb95462-5aed-4ee4-9223-06e45427a534","Type":"ContainerDied","Data":"3be08bf39cf012899b6acc1a68669c3770a334b45e0fd282ab67c682a6750dc4"}
Nov 03 07:45:18.830070 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.830015    4470 generic.go:356] "Generic (PLEG): container finished" podID="9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" containerID="a0b905d331b6f0ac173c904c8a6d34dbcc52281b09e30a5c42d4437c28883031" exitCode=0
Nov 03 07:45:18.830230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.830126    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc" event={"ID":"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c","Type":"ContainerDied","Data":"a0b905d331b6f0ac173c904c8a6d34dbcc52281b09e30a5c42d4437c28883031"}
Nov 03 07:45:18.830230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.830182    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc" event={"ID":"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c","Type":"ContainerDied","Data":"300f2c53d8b0fb10063f6b78489b8b9ee3931f07a5489447d2362c54b3e38ecc"}
Nov 03 07:45:18.830230 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.830204    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="300f2c53d8b0fb10063f6b78489b8b9ee3931f07a5489447d2362c54b3e38ecc"
Nov 03 07:45:18.832245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.832224    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz" event={"ID":"d93bbaf9-6594-4956-bc86-854e3c012f78","Type":"ContainerDied","Data":"e11b61638814d2fe6109696e597e0bcddcfa7608c8696f809c0316c55e83e3be"}
Nov 03 07:45:18.832436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.832321    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"
Nov 03 07:45:18.840691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.839754    4470 generic.go:356] "Generic (PLEG): container finished" podID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerID="39340b4b0bb650db47950c50dd59e315f89dd8f745ee067fb52053e308965f0a" exitCode=60
Nov 03 07:45:18.840691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.839832    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx" event={"ID":"cd4f0619-8935-4e73-83f9-5247d74f1897","Type":"ContainerDied","Data":"39340b4b0bb650db47950c50dd59e315f89dd8f745ee067fb52053e308965f0a"}
Nov 03 07:45:18.840691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.840228    4470 scope.go:122] "RemoveContainer" containerID="39340b4b0bb650db47950c50dd59e315f89dd8f745ee067fb52053e308965f0a"
Nov 03 07:45:18.840691 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:18.840508    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894865-p8jjx_odh-ai-gateway-infra(cd4f0619-8935-4e73-83f9-5247d74f1897)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897"
Nov 03 07:45:18.843516 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.843397    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2" event={"ID":"f39be689-c2f3-4091-9097-5729f65ae5c0","Type":"ContainerDied","Data":"70ef94786b5060d528882fc2c8e6e9e30d33303a0973aa2286eed309b6d13d89"}
Nov 03 07:45:18.843573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.843513    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"
Nov 03 07:45:18.844393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.844377    4470 scope.go:122] "RemoveContainer" containerID="17159b6e6cbe9644806522c207ffc1fb20435a65b7c1a19e3dab98b20bca02d9"
Nov 03 07:45:18.845795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.845624    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:45:18.863607 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.863574    4470 scope.go:122] "RemoveContainer" containerID="b1a8e6c21ca46709cba9697806e567099d0253e3ad33555ca021e73059100d7d"
Nov 03 07:45:18.878440 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.878413    4470 scope.go:122] "RemoveContainer" containerID="26162ed77ba097f7d1d281fb342694bb416819e94f66e0a1d3ca96f476252afb"
Nov 03 07:45:18.888548 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.888524    4470 scope.go:122] "RemoveContainer" containerID="c95a10208d653a9e9d87113cf77c0861f45a0c30d64f6ec426b5a213f84e132b"
Nov 03 07:45:18.901571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.901552    4470 scope.go:122] "RemoveContainer" containerID="360519caf0576b5975edb8034fc487a6c83ea28602901c5ba711e324ef428b14"
Nov 03 07:45:18.912826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.912801    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-token\" (UniqueName: \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-token\") pod \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") "
Nov 03 07:45:18.912928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.912860    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-kubeconfig\") pod \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") "
Nov 03 07:45:18.912928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.912911    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-dns-cert\") pod \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") "
Nov 03 07:45:18.913057 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.913002    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-ca-configmap\") pod \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") "
Nov 03 07:45:18.913057 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.913049    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-cacerts\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-cacerts\") pod \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") "
Nov 03 07:45:18.913194 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.913150    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-local-certs\") pod \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") "
Nov 03 07:45:18.913312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.913295    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-kube-api-access-kjfxl\" (UniqueName: \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-kube-api-access-kjfxl\") pod \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\" (UID: \"9bb31eb4-c728-4128-a2bd-a3ed99aaf99c\") "
Nov 03 07:45:18.913450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.913420    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-ca-configmap" pod "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" (UID: "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:18.913895 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.913874    4470 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-ca-configmap\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:18.915594 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.915555    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-kubeconfig" pod "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" (UID: "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:45:18.915994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.915967    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-dns-cert" pod "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" (UID: "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:45:18.916068 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.916007    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-local-certs" pod "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" (UID: "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:18.916068 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.916048    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-token" pod "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" (UID: "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:18.916242 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.916217    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-cacerts" pod "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" (UID: "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:45:18.916322 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:18.916310    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-kube-api-access-kjfxl" pod "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" (UID: "9bb31eb4-c728-4128-a2bd-a3ed99aaf99c"). InnerVolumeSpecName "kube-api-access-kjfxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:19.015020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.014942    4470 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-cacerts\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:19.015020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.014968    4470 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-local-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:19.015020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.014979    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjfxl\" (UniqueName: \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-kube-api-access-kjfxl\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:19.015020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.014988    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:19.015020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.014997    4470 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-kubeconfig\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:19.015020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.015008    4470 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c-istio-csr-dns-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:19.424747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.424710    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"]
Nov 03 07:45:19.433124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.433094    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-669d6b-29894865-swkq2"]
Nov 03 07:45:19.549160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.549102    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"]
Nov 03 07:45:19.554499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.554472    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50ae96-29894865-xvstp"]
Nov 03 07:45:19.587981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.587945    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"]
Nov 03 07:45:19.744966 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.744942    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"
Nov 03 07:45:19.788680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.788647    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"]
Nov 03 07:45:19.796389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.796360    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894865-nkvtj"]
Nov 03 07:45:19.810441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.810410    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f" path="/var/lib/kubelet/pods/698ba69f-9dc7-4a28-ab83-8791b51d938f/volumes"
Nov 03 07:45:19.810843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.810830    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71" path="/var/lib/kubelet/pods/76bf1202-fb69-421b-a075-7a5fdcec5f71/volumes"
Nov 03 07:45:19.811246 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.811233    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7" path="/var/lib/kubelet/pods/7bc1750b-bfa3-4e1b-a24f-a3ca122f01a7/volumes"
Nov 03 07:45:19.811875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.811856    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb95462-5aed-4ee4-9223-06e45427a534" path="/var/lib/kubelet/pods/7fb95462-5aed-4ee4-9223-06e45427a534/volumes"
Nov 03 07:45:19.823928 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.823902    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310e9a49-509b-4874-ac5e-dca0f8934fa2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/310e9a49-509b-4874-ac5e-dca0f8934fa2-maas-api-service-ca\") pod \"310e9a49-509b-4874-ac5e-dca0f8934fa2\" (UID: \"310e9a49-509b-4874-ac5e-dca0f8934fa2\") "
Nov 03 07:45:19.824274 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.824011    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310e9a49-509b-4874-ac5e-dca0f8934fa2-kube-api-access-lxpz6\" (UniqueName: \"kubernetes.io/projected/310e9a49-509b-4874-ac5e-dca0f8934fa2-kube-api-access-lxpz6\") pod \"310e9a49-509b-4874-ac5e-dca0f8934fa2\" (UID: \"310e9a49-509b-4874-ac5e-dca0f8934fa2\") "
Nov 03 07:45:19.824343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.824322    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310e9a49-509b-4874-ac5e-dca0f8934fa2-maas-api-service-ca" pod "310e9a49-509b-4874-ac5e-dca0f8934fa2" (UID: "310e9a49-509b-4874-ac5e-dca0f8934fa2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:19.824509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.824495    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/310e9a49-509b-4874-ac5e-dca0f8934fa2-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:19.826195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.826170    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310e9a49-509b-4874-ac5e-dca0f8934fa2-kube-api-access-lxpz6" pod "310e9a49-509b-4874-ac5e-dca0f8934fa2" (UID: "310e9a49-509b-4874-ac5e-dca0f8934fa2"). InnerVolumeSpecName "kube-api-access-lxpz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:19.863666 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.863639    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"
Nov 03 07:45:19.863823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.863661    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7" event={"ID":"310e9a49-509b-4874-ac5e-dca0f8934fa2","Type":"ContainerDied","Data":"193706051c165edb07e63e87f6345b3d59cf48e76c0cfcbed5267c5057363354"}
Nov 03 07:45:19.863823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.863722    4470 scope.go:122] "RemoveContainer" containerID="3c5666d8b1c028242c0a1053f062a038f5889410e57e35acfd5226a881990f87"
Nov 03 07:45:19.866851 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.866830    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"
Nov 03 07:45:19.910370 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.910342    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"]
Nov 03 07:45:19.916479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.916452    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1f2e7e-29894865-nbtkz"]
Nov 03 07:45:19.926159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:19.926114    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxpz6\" (UniqueName: \"kubernetes.io/projected/310e9a49-509b-4874-ac5e-dca0f8934fa2-kube-api-access-lxpz6\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:20.030631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.030548    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"]
Nov 03 07:45:20.033411 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.033390    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eac348-29894865-nf5q2"]
Nov 03 07:45:20.344970 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.344937    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"]
Nov 03 07:45:20.351573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.351545    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894865-c98c7"]
Nov 03 07:45:20.465989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.465954    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"]
Nov 03 07:45:20.472349 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.472323    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc"]
Nov 03 07:45:20.682370 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.682260    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"]
Nov 03 07:45:20.837791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.837767    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"
Nov 03 07:45:20.875455 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.875419    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"
Nov 03 07:45:20.875455 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.875447    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx" event={"ID":"cd4f0619-8935-4e73-83f9-5247d74f1897","Type":"ContainerDied","Data":"97a53d29e3cb6d4828d07fb018e8fe26c521ccd93361da253f1354b7e6c3b1dc"}
Nov 03 07:45:20.875686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.875513    4470 scope.go:122] "RemoveContainer" containerID="39340b4b0bb650db47950c50dd59e315f89dd8f745ee067fb52053e308965f0a"
Nov 03 07:45:20.937455 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.937381    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4f0619-8935-4e73-83f9-5247d74f1897-kube-api-access-lfqcq\" (UniqueName: \"kubernetes.io/projected/cd4f0619-8935-4e73-83f9-5247d74f1897-kube-api-access-lfqcq\") pod \"cd4f0619-8935-4e73-83f9-5247d74f1897\" (UID: \"cd4f0619-8935-4e73-83f9-5247d74f1897\") "
Nov 03 07:45:20.937625 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.937474    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd4f0619-8935-4e73-83f9-5247d74f1897-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd4f0619-8935-4e73-83f9-5247d74f1897-maas-api-service-ca\") pod \"cd4f0619-8935-4e73-83f9-5247d74f1897\" (UID: \"cd4f0619-8935-4e73-83f9-5247d74f1897\") "
Nov 03 07:45:20.937861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.937838    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4f0619-8935-4e73-83f9-5247d74f1897-maas-api-service-ca" pod "cd4f0619-8935-4e73-83f9-5247d74f1897" (UID: "cd4f0619-8935-4e73-83f9-5247d74f1897"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:20.938030 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.938015    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd4f0619-8935-4e73-83f9-5247d74f1897-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:20.939553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:20.939526    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4f0619-8935-4e73-83f9-5247d74f1897-kube-api-access-lfqcq" pod "cd4f0619-8935-4e73-83f9-5247d74f1897" (UID: "cd4f0619-8935-4e73-83f9-5247d74f1897"). InnerVolumeSpecName "kube-api-access-lfqcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:21.039658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:21.039626    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfqcq\" (UniqueName: \"kubernetes.io/projected/cd4f0619-8935-4e73-83f9-5247d74f1897-kube-api-access-lfqcq\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:21.307702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:21.307607    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"]
Nov 03 07:45:21.313577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:21.313550    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894865-p8jjx"]
Nov 03 07:45:21.807372 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:21.807336    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2" path="/var/lib/kubelet/pods/310e9a49-509b-4874-ac5e-dca0f8934fa2/volumes"
Nov 03 07:45:21.808050 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:21.808031    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" path="/var/lib/kubelet/pods/9bb31eb4-c728-4128-a2bd-a3ed99aaf99c/volumes"
Nov 03 07:45:21.808568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:21.808551    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897" path="/var/lib/kubelet/pods/cd4f0619-8935-4e73-83f9-5247d74f1897/volumes"
Nov 03 07:45:21.809323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:21.809307    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78" path="/var/lib/kubelet/pods/d93bbaf9-6594-4956-bc86-854e3c012f78/volumes"
Nov 03 07:45:21.809676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:21.809664    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0" path="/var/lib/kubelet/pods/f39be689-c2f3-4091-9097-5729f65ae5c0/volumes"
Nov 03 07:45:25.384111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384071    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"]
Nov 03 07:45:25.384631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384617    4470 cpu_manager.go:414] "containerMap: removing container" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerName="cleanup"
Nov 03 07:45:25.384680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384632    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerName="cleanup"
Nov 03 07:45:25.384680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384642    4470 cpu_manager.go:414] "containerMap: removing container" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerName="cleanup"
Nov 03 07:45:25.384680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384647    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerName="cleanup"
Nov 03 07:45:25.384680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384655    4470 cpu_manager.go:414] "containerMap: removing container" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerName="cleanup"
Nov 03 07:45:25.384680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384660    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerName="cleanup"
Nov 03 07:45:25.384680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384667    4470 cpu_manager.go:414] "containerMap: removing container" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerName="cleanup"
Nov 03 07:45:25.384680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384671    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerName="cleanup"
Nov 03 07:45:25.384680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384678    4470 cpu_manager.go:414] "containerMap: removing container" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerName="cleanup"
Nov 03 07:45:25.384680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384683    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384698    4470 cpu_manager.go:414] "containerMap: removing container" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384702    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384709    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7fb95462-5aed-4ee4-9223-06e45427a534" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384714    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb95462-5aed-4ee4-9223-06e45427a534" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384721    4470 cpu_manager.go:414] "containerMap: removing container" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384725    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384731    4470 cpu_manager.go:414] "containerMap: removing container" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384736    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384745    4470 cpu_manager.go:414] "containerMap: removing container" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384749    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384756    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7fb95462-5aed-4ee4-9223-06e45427a534" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384760    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb95462-5aed-4ee4-9223-06e45427a534" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384768    4470 cpu_manager.go:414] "containerMap: removing container" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384771    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384778    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384782    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384789    4470 cpu_manager.go:414] "containerMap: removing container" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384794    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384803    4470 cpu_manager.go:414] "containerMap: removing container" podUID="9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" containerName="discovery"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384807    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" containerName="discovery"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384817    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7fb95462-5aed-4ee4-9223-06e45427a534" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384821    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb95462-5aed-4ee4-9223-06e45427a534" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384837    4470 cpu_manager.go:414] "containerMap: removing container" podUID="59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" containerName="istio-proxy"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384841    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" containerName="istio-proxy"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384848    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384851    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384949    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384957    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384964    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb95462-5aed-4ee4-9223-06e45427a534" containerName="cleanup"
Nov 03 07:45:25.384955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384970    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384975    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384980    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384988    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb95462-5aed-4ee4-9223-06e45427a534" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384993    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.384998    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.385003    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.385009    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb31eb4-c728-4128-a2bd-a3ed99aaf99c" containerName="discovery"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.385014    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.385018    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb95462-5aed-4ee4-9223-06e45427a534" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.385025    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.385030    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.385034    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerName="cleanup"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.385039    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="59dbf6a0-8d6e-4cbc-bde3-7b665e9021df" containerName="istio-proxy"
Nov 03 07:45:25.385885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.385043    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerName="cleanup"
Nov 03 07:45:25.389982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.389964    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:25.448774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.448738    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"]
Nov 03 07:45:25.493236 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.493200    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d47bf7-dlqlr\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:25.493236 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.493238    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d47bf7-dlqlr\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:25.493481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.493356    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kx5v\" (UniqueName: \"kubernetes.io/projected/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-kube-api-access-6kx5v\") pod \"maas-api-revoke-sub-c3d47bf7-dlqlr\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:25.495636 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.495619    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:45:25.495754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.495708    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:45:25.495838 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.495824    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:45:25.496767 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.496755    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:45:25.594393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.594349    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d47bf7-dlqlr\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:25.594393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.594397    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d47bf7-dlqlr\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:25.594616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.594453    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kx5v\" (UniqueName: \"kubernetes.io/projected/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-kube-api-access-6kx5v\") pod \"maas-api-revoke-sub-c3d47bf7-dlqlr\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:25.594991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.594965    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d47bf7-dlqlr\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:25.596914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.596887    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d47bf7-dlqlr\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:25.659595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.659510    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kx5v\" (UniqueName: \"kubernetes.io/projected/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-kube-api-access-6kx5v\") pod \"maas-api-revoke-sub-c3d47bf7-dlqlr\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:25.703254 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:25.703221    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:26.081224 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:26.081195    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"]
Nov 03 07:45:26.082293 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:45:26.082266    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7702a8_fd0f_44f0_a293_6d9980c7fc73.slice/crio-80e16a7c9dd43be88fcdb703b5256036d3b0bce86cb60c8903e9b5dfa6e88031 WatchSource:0}: Error finding container 80e16a7c9dd43be88fcdb703b5256036d3b0bce86cb60c8903e9b5dfa6e88031: Status 404 returned error can't find the container with id 80e16a7c9dd43be88fcdb703b5256036d3b0bce86cb60c8903e9b5dfa6e88031
Nov 03 07:45:26.920614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:26.920580    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr" event={"ID":"7f7702a8-fd0f-44f0-a293-6d9980c7fc73","Type":"ContainerStarted","Data":"9c1e246a28bc18fe4738090959aac847104136ed03eb3245432377ede3ff2d2b"}
Nov 03 07:45:26.921064 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:26.920642    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr" event={"ID":"7f7702a8-fd0f-44f0-a293-6d9980c7fc73","Type":"ContainerStarted","Data":"80e16a7c9dd43be88fcdb703b5256036d3b0bce86cb60c8903e9b5dfa6e88031"}
Nov 03 07:45:27.041306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:27.041255    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr" podStartSLOduration=2.041238887 podStartE2EDuration="2.041238887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:27.040041401 +0000 UTC m=+2779.372410394" watchObservedRunningTime="2026-11-03 07:45:27.041238887 +0000 UTC m=+2779.373607880"
Nov 03 07:45:27.928759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:27.928724    4470 generic.go:356] "Generic (PLEG): container finished" podID="7f7702a8-fd0f-44f0-a293-6d9980c7fc73" containerID="9c1e246a28bc18fe4738090959aac847104136ed03eb3245432377ede3ff2d2b" exitCode=0
Nov 03 07:45:27.929171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:27.928805    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr" event={"ID":"7f7702a8-fd0f-44f0-a293-6d9980c7fc73","Type":"ContainerDied","Data":"9c1e246a28bc18fe4738090959aac847104136ed03eb3245432377ede3ff2d2b"}
Nov 03 07:45:29.089939 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.089912    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:29.138843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.138807    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-xks-ca\") pod \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") "
Nov 03 07:45:29.139045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.138879    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-openshift-ca\") pod \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") "
Nov 03 07:45:29.139045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.138971    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-kube-api-access-6kx5v\" (UniqueName: \"kubernetes.io/projected/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-kube-api-access-6kx5v\") pod \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\" (UID: \"7f7702a8-fd0f-44f0-a293-6d9980c7fc73\") "
Nov 03 07:45:29.139325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.139289    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-openshift-ca" pod "7f7702a8-fd0f-44f0-a293-6d9980c7fc73" (UID: "7f7702a8-fd0f-44f0-a293-6d9980c7fc73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:29.139527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.139508    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:29.141407 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.141377    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-kube-api-access-6kx5v" pod "7f7702a8-fd0f-44f0-a293-6d9980c7fc73" (UID: "7f7702a8-fd0f-44f0-a293-6d9980c7fc73"). InnerVolumeSpecName "kube-api-access-6kx5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:29.141639 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.141611    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-xks-ca" pod "7f7702a8-fd0f-44f0-a293-6d9980c7fc73" (UID: "7f7702a8-fd0f-44f0-a293-6d9980c7fc73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:45:29.142947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.142925    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"]
Nov 03 07:45:29.191849 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.191770    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18b89b-8cv9t"]
Nov 03 07:45:29.241045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.241009    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:29.241045 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.241041    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kx5v\" (UniqueName: \"kubernetes.io/projected/7f7702a8-fd0f-44f0-a293-6d9980c7fc73-kube-api-access-6kx5v\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:29.806047 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.806011    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b134e69-9c3b-43c8-b418-2a845681536b" path="/var/lib/kubelet/pods/8b134e69-9c3b-43c8-b418-2a845681536b/volumes"
Nov 03 07:45:29.944265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.944231    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"
Nov 03 07:45:29.944443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.944267    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr" event={"ID":"7f7702a8-fd0f-44f0-a293-6d9980c7fc73","Type":"ContainerDied","Data":"80e16a7c9dd43be88fcdb703b5256036d3b0bce86cb60c8903e9b5dfa6e88031"}
Nov 03 07:45:29.944443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:29.944314    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80e16a7c9dd43be88fcdb703b5256036d3b0bce86cb60c8903e9b5dfa6e88031"
Nov 03 07:45:34.759431 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:34.759396    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:45:35.456703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:35.456670    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"]
Nov 03 07:45:35.507105 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:35.507071    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44c08f-q6cbb"]
Nov 03 07:45:35.807422 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:35.807348    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948812f6-4c5e-461b-b6a2-4d697f16c48f" path="/var/lib/kubelet/pods/948812f6-4c5e-461b-b6a2-4d697f16c48f/volumes"
Nov 03 07:45:37.645925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:37.645887    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"]
Nov 03 07:45:37.646453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:37.646193    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy" containerID="cri-o://c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483" gracePeriod=30
Nov 03 07:45:38.674691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.674646    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"]
Nov 03 07:45:38.675415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675394    4470 cpu_manager.go:414] "containerMap: removing container" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerName="cleanup"
Nov 03 07:45:38.675512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675415    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerName="cleanup"
Nov 03 07:45:38.675512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675437    4470 cpu_manager.go:414] "containerMap: removing container" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerName="cleanup"
Nov 03 07:45:38.675512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675444    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerName="cleanup"
Nov 03 07:45:38.675512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675461    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7f7702a8-fd0f-44f0-a293-6d9980c7fc73" containerName="revoke-keys"
Nov 03 07:45:38.675512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675468    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7702a8-fd0f-44f0-a293-6d9980c7fc73" containerName="revoke-keys"
Nov 03 07:45:38.675512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675481    4470 cpu_manager.go:414] "containerMap: removing container" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerName="cleanup"
Nov 03 07:45:38.675512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675489    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerName="cleanup"
Nov 03 07:45:38.675512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675508    4470 cpu_manager.go:414] "containerMap: removing container" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerName="cleanup"
Nov 03 07:45:38.676004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675515    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerName="cleanup"
Nov 03 07:45:38.676004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675529    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerName="cleanup"
Nov 03 07:45:38.676004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675536    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93bbaf9-6594-4956-bc86-854e3c012f78" containerName="cleanup"
Nov 03 07:45:38.676004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675685    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bf1202-fb69-421b-a075-7a5fdcec5f71" containerName="cleanup"
Nov 03 07:45:38.676004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675705    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="310e9a49-509b-4874-ac5e-dca0f8934fa2" containerName="cleanup"
Nov 03 07:45:38.676004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675714    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7702a8-fd0f-44f0-a293-6d9980c7fc73" containerName="revoke-keys"
Nov 03 07:45:38.676004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675724    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="f39be689-c2f3-4091-9097-5729f65ae5c0" containerName="cleanup"
Nov 03 07:45:38.676004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675733    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4f0619-8935-4e73-83f9-5247d74f1897" containerName="cleanup"
Nov 03 07:45:38.676004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.675748    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="698ba69f-9dc7-4a28-ab83-8791b51d938f" containerName="cleanup"
Nov 03 07:45:38.679582 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.679563    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:38.697353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.697321    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"]
Nov 03 07:45:38.701252 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.701236    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:38.739633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.739598    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:45:38.742181 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.742157    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"]
Nov 03 07:45:38.744962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.744942    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:45:38.760965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.760930    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfw8m\" (UniqueName: \"kubernetes.io/projected/7c425ee9-4286-4ab3-a737-c01ab814de0e-kube-api-access-pfw8m\") pod \"maas-api-revoke-sub-4158a54e-m4tjv\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:38.761089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.760979    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:38.761214 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.761188    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4158a54e-m4tjv\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:38.761358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.761307    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fdjs\" (UniqueName: \"kubernetes.io/projected/aa8ceb69-165f-4754-9df9-4da9496cc82e-kube-api-access-5fdjs\") pod \"maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:38.761483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.761406    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:38.761483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.761466    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4158a54e-m4tjv\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:38.807468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.807431    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"]
Nov 03 07:45:38.828934 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.828901    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:45:38.829080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.828973    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:45:38.862452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.862415    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4158a54e-m4tjv\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:38.862631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.862470    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fdjs\" (UniqueName: \"kubernetes.io/projected/aa8ceb69-165f-4754-9df9-4da9496cc82e-kube-api-access-5fdjs\") pod \"maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:38.862631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.862515    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:38.862747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.862677    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4158a54e-m4tjv\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:38.862855 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.862834    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfw8m\" (UniqueName: \"kubernetes.io/projected/7c425ee9-4286-4ab3-a737-c01ab814de0e-kube-api-access-pfw8m\") pod \"maas-api-revoke-sub-4158a54e-m4tjv\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:38.862929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.862870    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:38.863496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.863469    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4158a54e-m4tjv\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:38.863593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.863570    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:38.865689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.865663    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:38.865774 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.865666    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4158a54e-m4tjv\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:38.926710 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.926641    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fdjs\" (UniqueName: \"kubernetes.io/projected/aa8ceb69-165f-4754-9df9-4da9496cc82e-kube-api-access-5fdjs\") pod \"maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:38.929872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.929845    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfw8m\" (UniqueName: \"kubernetes.io/projected/7c425ee9-4286-4ab3-a737-c01ab814de0e-kube-api-access-pfw8m\") pod \"maas-api-revoke-sub-4158a54e-m4tjv\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:38.993393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:38.993363    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:39.014915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:39.014887    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:39.379304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:39.379266    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"]
Nov 03 07:45:39.397219 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:45:39.397192    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c425ee9_4286_4ab3_a737_c01ab814de0e.slice/crio-a902f4a0a0b4aaf3b695ee4b7785ee8606d4b6f76237d74fc6337557ab5c16c0 WatchSource:0}: Error finding container a902f4a0a0b4aaf3b695ee4b7785ee8606d4b6f76237d74fc6337557ab5c16c0: Status 404 returned error can't find the container with id a902f4a0a0b4aaf3b695ee4b7785ee8606d4b6f76237d74fc6337557ab5c16c0
Nov 03 07:45:39.427722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:39.427698    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"]
Nov 03 07:45:39.435468 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:45:39.435431    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8ceb69_165f_4754_9df9_4da9496cc82e.slice/crio-0781d04696ad8ff2d68a80615438541d257c377b45075365ff6cf954ac7a0df5 WatchSource:0}: Error finding container 0781d04696ad8ff2d68a80615438541d257c377b45075365ff6cf954ac7a0df5: Status 404 returned error can't find the container with id 0781d04696ad8ff2d68a80615438541d257c377b45075365ff6cf954ac7a0df5
Nov 03 07:45:40.012327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:40.012233    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr" event={"ID":"aa8ceb69-165f-4754-9df9-4da9496cc82e","Type":"ContainerStarted","Data":"d6ac8b94ceb3866c529535b7c4b21c4abf57485e26c2c658632605f1ce2edcac"}
Nov 03 07:45:40.012327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:40.012292    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr" event={"ID":"aa8ceb69-165f-4754-9df9-4da9496cc82e","Type":"ContainerStarted","Data":"0781d04696ad8ff2d68a80615438541d257c377b45075365ff6cf954ac7a0df5"}
Nov 03 07:45:40.013992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:40.013965    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv" event={"ID":"7c425ee9-4286-4ab3-a737-c01ab814de0e","Type":"ContainerStarted","Data":"ee79d59b6f9152bce69f7add9e95aa3b16edb9f67f3d9084d5d0e94362a549a0"}
Nov 03 07:45:40.014097 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:40.014018    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv" event={"ID":"7c425ee9-4286-4ab3-a737-c01ab814de0e","Type":"ContainerStarted","Data":"a902f4a0a0b4aaf3b695ee4b7785ee8606d4b6f76237d74fc6337557ab5c16c0"}
Nov 03 07:45:40.136480 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:40.136418    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr" podStartSLOduration=2.136405686 podStartE2EDuration="2.136405686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:40.135527543 +0000 UTC m=+2792.467896538" watchObservedRunningTime="2026-11-03 07:45:40.136405686 +0000 UTC m=+2792.468774716"
Nov 03 07:45:40.255374 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:40.255243    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv" podStartSLOduration=2.255218667 podStartE2EDuration="2.255218667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:40.255096508 +0000 UTC m=+2792.587465501" watchObservedRunningTime="2026-11-03 07:45:40.255218667 +0000 UTC m=+2792.587587662"
Nov 03 07:45:41.021818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:41.021784    4470 generic.go:356] "Generic (PLEG): container finished" podID="aa8ceb69-165f-4754-9df9-4da9496cc82e" containerID="d6ac8b94ceb3866c529535b7c4b21c4abf57485e26c2c658632605f1ce2edcac" exitCode=0
Nov 03 07:45:41.022373 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:41.021863    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr" event={"ID":"aa8ceb69-165f-4754-9df9-4da9496cc82e","Type":"ContainerDied","Data":"d6ac8b94ceb3866c529535b7c4b21c4abf57485e26c2c658632605f1ce2edcac"}
Nov 03 07:45:41.023427 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:41.023405    4470 generic.go:356] "Generic (PLEG): container finished" podID="7c425ee9-4286-4ab3-a737-c01ab814de0e" containerID="ee79d59b6f9152bce69f7add9e95aa3b16edb9f67f3d9084d5d0e94362a549a0" exitCode=0
Nov 03 07:45:41.023512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:41.023489    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv" event={"ID":"7c425ee9-4286-4ab3-a737-c01ab814de0e","Type":"ContainerDied","Data":"ee79d59b6f9152bce69f7add9e95aa3b16edb9f67f3d9084d5d0e94362a549a0"}
Nov 03 07:45:41.823222 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:41.823185    4470 patch_prober.go:29] interesting pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:45:41.823405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:41.823250    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:45:42.210656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.210627    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:42.214752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.214720    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:42.304110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.304074    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-openshift-ca\") pod \"7c425ee9-4286-4ab3-a737-c01ab814de0e\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") "
Nov 03 07:45:42.304343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.304120    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c425ee9-4286-4ab3-a737-c01ab814de0e-kube-api-access-pfw8m\" (UniqueName: \"kubernetes.io/projected/7c425ee9-4286-4ab3-a737-c01ab814de0e-kube-api-access-pfw8m\") pod \"7c425ee9-4286-4ab3-a737-c01ab814de0e\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") "
Nov 03 07:45:42.304343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.304185    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8ceb69-165f-4754-9df9-4da9496cc82e-kube-api-access-5fdjs\" (UniqueName: \"kubernetes.io/projected/aa8ceb69-165f-4754-9df9-4da9496cc82e-kube-api-access-5fdjs\") pod \"aa8ceb69-165f-4754-9df9-4da9496cc82e\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") "
Nov 03 07:45:42.304343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.304216    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-openshift-ca\") pod \"aa8ceb69-165f-4754-9df9-4da9496cc82e\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") "
Nov 03 07:45:42.304343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.304244    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-xks-ca\") pod \"7c425ee9-4286-4ab3-a737-c01ab814de0e\" (UID: \"7c425ee9-4286-4ab3-a737-c01ab814de0e\") "
Nov 03 07:45:42.304343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.304293    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-xks-ca\") pod \"aa8ceb69-165f-4754-9df9-4da9496cc82e\" (UID: \"aa8ceb69-165f-4754-9df9-4da9496cc82e\") "
Nov 03 07:45:42.304634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.304532    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-openshift-ca" pod "7c425ee9-4286-4ab3-a737-c01ab814de0e" (UID: "7c425ee9-4286-4ab3-a737-c01ab814de0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:42.304634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.304595    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-openshift-ca" pod "aa8ceb69-165f-4754-9df9-4da9496cc82e" (UID: "aa8ceb69-165f-4754-9df9-4da9496cc82e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:42.305154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.305104    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:42.305154 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.305153    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:42.306624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.306596    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-xks-ca" pod "7c425ee9-4286-4ab3-a737-c01ab814de0e" (UID: "7c425ee9-4286-4ab3-a737-c01ab814de0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:45:42.306717 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.306686    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-xks-ca" pod "aa8ceb69-165f-4754-9df9-4da9496cc82e" (UID: "aa8ceb69-165f-4754-9df9-4da9496cc82e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:45:42.306771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.306714    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8ceb69-165f-4754-9df9-4da9496cc82e-kube-api-access-5fdjs" pod "aa8ceb69-165f-4754-9df9-4da9496cc82e" (UID: "aa8ceb69-165f-4754-9df9-4da9496cc82e"). InnerVolumeSpecName "kube-api-access-5fdjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:42.306771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.306744    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c425ee9-4286-4ab3-a737-c01ab814de0e-kube-api-access-pfw8m" pod "7c425ee9-4286-4ab3-a737-c01ab814de0e" (UID: "7c425ee9-4286-4ab3-a737-c01ab814de0e"). InnerVolumeSpecName "kube-api-access-pfw8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:42.406586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.406510    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfw8m\" (UniqueName: \"kubernetes.io/projected/7c425ee9-4286-4ab3-a737-c01ab814de0e-kube-api-access-pfw8m\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:42.406586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.406537    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fdjs\" (UniqueName: \"kubernetes.io/projected/aa8ceb69-165f-4754-9df9-4da9496cc82e-kube-api-access-5fdjs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:42.406586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.406549    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c425ee9-4286-4ab3-a737-c01ab814de0e-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:42.406586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.406558    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa8ceb69-165f-4754-9df9-4da9496cc82e-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:42.899021 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:42.898994    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:45:43.012481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012395    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-socket\") pod \"d336591b-65cf-4a16-a050-569193c8f481\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") "
Nov 03 07:45:43.012481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012478    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d336591b-65cf-4a16-a050-569193c8f481-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d336591b-65cf-4a16-a050-569193c8f481-istio-podinfo\") pod \"d336591b-65cf-4a16-a050-569193c8f481\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") "
Nov 03 07:45:43.012727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012509    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-data\") pod \"d336591b-65cf-4a16-a050-569193c8f481\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") "
Nov 03 07:45:43.012727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012541    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-envoy\") pod \"d336591b-65cf-4a16-a050-569193c8f481\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") "
Nov 03 07:45:43.012727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012566    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-certs\") pod \"d336591b-65cf-4a16-a050-569193c8f481\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") "
Nov 03 07:45:43.012727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012611    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-credential-socket\") pod \"d336591b-65cf-4a16-a050-569193c8f481\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") "
Nov 03 07:45:43.012727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012691    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d336591b-65cf-4a16-a050-569193c8f481-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d336591b-65cf-4a16-a050-569193c8f481-istiod-ca-cert\") pod \"d336591b-65cf-4a16-a050-569193c8f481\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") "
Nov 03 07:45:43.012727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012703    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-socket" pod "d336591b-65cf-4a16-a050-569193c8f481" (UID: "d336591b-65cf-4a16-a050-569193c8f481"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:43.012727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012720    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-kube-api-access-n8v8f\" (UniqueName: \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-kube-api-access-n8v8f\") pod \"d336591b-65cf-4a16-a050-569193c8f481\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") "
Nov 03 07:45:43.013077 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012766    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-istio-token\" (UniqueName: \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-istio-token\") pod \"d336591b-65cf-4a16-a050-569193c8f481\" (UID: \"d336591b-65cf-4a16-a050-569193c8f481\") "
Nov 03 07:45:43.013077 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012930    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-certs" pod "d336591b-65cf-4a16-a050-569193c8f481" (UID: "d336591b-65cf-4a16-a050-569193c8f481"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:43.013077 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012912    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-data" pod "d336591b-65cf-4a16-a050-569193c8f481" (UID: "d336591b-65cf-4a16-a050-569193c8f481"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:43.013077 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.012975    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-credential-socket" pod "d336591b-65cf-4a16-a050-569193c8f481" (UID: "d336591b-65cf-4a16-a050-569193c8f481"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:43.013296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.013206    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d336591b-65cf-4a16-a050-569193c8f481-istiod-ca-cert" pod "d336591b-65cf-4a16-a050-569193c8f481" (UID: "d336591b-65cf-4a16-a050-569193c8f481"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:43.013426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.013405    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:43.013500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.013435    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d336591b-65cf-4a16-a050-569193c8f481-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:43.013500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.013451    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:43.013789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.013487    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:43.013789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.013609    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:43.015111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.015085    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-envoy" pod "d336591b-65cf-4a16-a050-569193c8f481" (UID: "d336591b-65cf-4a16-a050-569193c8f481"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:43.015245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.015115    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-kube-api-access-n8v8f" pod "d336591b-65cf-4a16-a050-569193c8f481" (UID: "d336591b-65cf-4a16-a050-569193c8f481"). InnerVolumeSpecName "kube-api-access-n8v8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:43.015245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.015162    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d336591b-65cf-4a16-a050-569193c8f481-istio-podinfo" pod "d336591b-65cf-4a16-a050-569193c8f481" (UID: "d336591b-65cf-4a16-a050-569193c8f481"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:45:43.015245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.015171    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-istio-token" pod "d336591b-65cf-4a16-a050-569193c8f481" (UID: "d336591b-65cf-4a16-a050-569193c8f481"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:43.039891 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.039854    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"
Nov 03 07:45:43.039891 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.039868    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr" event={"ID":"aa8ceb69-165f-4754-9df9-4da9496cc82e","Type":"ContainerDied","Data":"0781d04696ad8ff2d68a80615438541d257c377b45075365ff6cf954ac7a0df5"}
Nov 03 07:45:43.040106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.039928    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0781d04696ad8ff2d68a80615438541d257c377b45075365ff6cf954ac7a0df5"
Nov 03 07:45:43.046507 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.046484    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"
Nov 03 07:45:43.046655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.046529    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv" event={"ID":"7c425ee9-4286-4ab3-a737-c01ab814de0e","Type":"ContainerDied","Data":"a902f4a0a0b4aaf3b695ee4b7785ee8606d4b6f76237d74fc6337557ab5c16c0"}
Nov 03 07:45:43.046655 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.046585    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a902f4a0a0b4aaf3b695ee4b7785ee8606d4b6f76237d74fc6337557ab5c16c0"
Nov 03 07:45:43.048533 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.048508    4470 generic.go:356] "Generic (PLEG): container finished" podID="d336591b-65cf-4a16-a050-569193c8f481" containerID="c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483" exitCode=0
Nov 03 07:45:43.048676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.048559    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" event={"ID":"d336591b-65cf-4a16-a050-569193c8f481","Type":"ContainerDied","Data":"c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483"}
Nov 03 07:45:43.048676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.048597    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"
Nov 03 07:45:43.048676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.048613    4470 scope.go:122] "RemoveContainer" containerID="c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483"
Nov 03 07:45:43.048676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.048603    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp" event={"ID":"d336591b-65cf-4a16-a050-569193c8f481","Type":"ContainerDied","Data":"def44a8c050e53da15debdb6c93bfc67cb1249684450f5b726c1fab00fc0f738"}
Nov 03 07:45:43.061624 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.061602    4470 scope.go:122] "RemoveContainer" containerID="cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d"
Nov 03 07:45:43.071559 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.071541    4470 scope.go:122] "RemoveContainer" containerID="c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483"
Nov 03 07:45:43.071881 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:43.071861    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483\": container with ID starting with c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483 not found: ID does not exist" containerID="c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483"
Nov 03 07:45:43.071948 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.071900    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483"} err="failed to get container status \"c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483\": rpc error: code = NotFound desc = could not find container \"c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483\": container with ID starting with c87699372e90756e78be96f10cd499f196f95ecb5e215d6ce85c88b8311e5483 not found: ID does not exist"
Nov 03 07:45:43.071948 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.071920    4470 scope.go:122] "RemoveContainer" containerID="cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d"
Nov 03 07:45:43.072236 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:43.072218    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d\": container with ID starting with cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d not found: ID does not exist" containerID="cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d"
Nov 03 07:45:43.072298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.072243    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d"} err="failed to get container status \"cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d\": rpc error: code = NotFound desc = could not find container \"cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d\": container with ID starting with cb235bd20986e44ba1c5aac5a10df42d139f4c23e5cf6e5f9b3bd05ccebc482d not found: ID does not exist"
Nov 03 07:45:43.114585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.114556    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:43.114585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.114583    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d336591b-65cf-4a16-a050-569193c8f481-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:43.114786 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.114594    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d336591b-65cf-4a16-a050-569193c8f481-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:43.114786 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.114605    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8v8f\" (UniqueName: \"kubernetes.io/projected/d336591b-65cf-4a16-a050-569193c8f481-kube-api-access-n8v8f\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:43.142396 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.142358    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"]
Nov 03 07:45:43.332976 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.332940    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"]
Nov 03 07:45:43.343884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.343858    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp"]
Nov 03 07:45:43.811096 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.811009    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d336591b-65cf-4a16-a050-569193c8f481" path="/var/lib/kubelet/pods/d336591b-65cf-4a16-a050-569193c8f481/volumes"
Nov 03 07:45:43.943676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.943638    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"]
Nov 03 07:45:43.943869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:43.943848    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy" containerID="cri-o://92d25661d9056277ba97bfb36c8b9ef9ee2ee56eb850278fdbc53e3c2ac80c96" gracePeriod=30
Nov 03 07:45:44.986101 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986063    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"]
Nov 03 07:45:44.986671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986652    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7c425ee9-4286-4ab3-a737-c01ab814de0e" containerName="revoke-keys"
Nov 03 07:45:44.986747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986674    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c425ee9-4286-4ab3-a737-c01ab814de0e" containerName="revoke-keys"
Nov 03 07:45:44.986747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986699    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy"
Nov 03 07:45:44.986747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986704    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy"
Nov 03 07:45:44.986747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986714    4470 cpu_manager.go:414] "containerMap: removing container" podUID="aa8ceb69-165f-4754-9df9-4da9496cc82e" containerName="revoke-keys"
Nov 03 07:45:44.986747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986719    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8ceb69-165f-4754-9df9-4da9496cc82e" containerName="revoke-keys"
Nov 03 07:45:44.986747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986729    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy"
Nov 03 07:45:44.986747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986733    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy"
Nov 03 07:45:44.987127 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986825    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c425ee9-4286-4ab3-a737-c01ab814de0e" containerName="revoke-keys"
Nov 03 07:45:44.987127 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986835    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy"
Nov 03 07:45:44.987127 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986842    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8ceb69-165f-4754-9df9-4da9496cc82e" containerName="revoke-keys"
Nov 03 07:45:44.987127 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.986848    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d336591b-65cf-4a16-a050-569193c8f481" containerName="istio-proxy"
Nov 03 07:45:44.991579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:44.991558    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:45.048947 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.048912    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"]
Nov 03 07:45:45.096215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.096177    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:45:45.096215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.096177    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:45:45.096449 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.096251    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:45:45.096449 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.096296    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:45:45.138548 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.138517    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:45.138728 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.138652    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckn2j\" (UniqueName: \"kubernetes.io/projected/7ff1e722-5ad1-448c-9339-a26551fe84da-kube-api-access-ckn2j\") pod \"maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:45.138850 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.138835    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:45.240298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.240223    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckn2j\" (UniqueName: \"kubernetes.io/projected/7ff1e722-5ad1-448c-9339-a26551fe84da-kube-api-access-ckn2j\") pod \"maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:45.240462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.240331    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:45.240462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.240392    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:45.241016 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.240988    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:45.242925 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.242905    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:45.304152 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.304111    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckn2j\" (UniqueName: \"kubernetes.io/projected/7ff1e722-5ad1-448c-9339-a26551fe84da-kube-api-access-ckn2j\") pod \"maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:45.306971 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.306952    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:45.691850 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:45.691819    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"]
Nov 03 07:45:45.692906 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:45:45.692875    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff1e722_5ad1_448c_9339_a26551fe84da.slice/crio-67a3dcb11fbfebbc3167722e285fdee5590fcfd7baa0cad34c44ba34256facd6 WatchSource:0}: Error finding container 67a3dcb11fbfebbc3167722e285fdee5590fcfd7baa0cad34c44ba34256facd6: Status 404 returned error can't find the container with id 67a3dcb11fbfebbc3167722e285fdee5590fcfd7baa0cad34c44ba34256facd6
Nov 03 07:45:46.103971 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:46.103931    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm" event={"ID":"7ff1e722-5ad1-448c-9339-a26551fe84da","Type":"ContainerStarted","Data":"d948b1489588309c554ae901aa550dd09a4a08b16d5946c808afb174a9357614"}
Nov 03 07:45:46.104397 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:46.103998    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm" event={"ID":"7ff1e722-5ad1-448c-9339-a26551fe84da","Type":"ContainerStarted","Data":"67a3dcb11fbfebbc3167722e285fdee5590fcfd7baa0cad34c44ba34256facd6"}
Nov 03 07:45:46.225014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:46.224896    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm" podStartSLOduration=2.224876903 podStartE2EDuration="2.224876903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:46.22283472 +0000 UTC m=+2798.555203714" watchObservedRunningTime="2026-11-03 07:45:46.224876903 +0000 UTC m=+2798.557245898"
Nov 03 07:45:47.111691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:47.111660    4470 generic.go:356] "Generic (PLEG): container finished" podID="7ff1e722-5ad1-448c-9339-a26551fe84da" containerID="d948b1489588309c554ae901aa550dd09a4a08b16d5946c808afb174a9357614" exitCode=0
Nov 03 07:45:47.112215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:47.111740    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm" event={"ID":"7ff1e722-5ad1-448c-9339-a26551fe84da","Type":"ContainerDied","Data":"d948b1489588309c554ae901aa550dd09a4a08b16d5946c808afb174a9357614"}
Nov 03 07:45:48.272685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:48.272660    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:48.378274 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:48.378195    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-xks-ca\") pod \"7ff1e722-5ad1-448c-9339-a26551fe84da\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") "
Nov 03 07:45:48.378274 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:48.378231    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff1e722-5ad1-448c-9339-a26551fe84da-kube-api-access-ckn2j\" (UniqueName: \"kubernetes.io/projected/7ff1e722-5ad1-448c-9339-a26551fe84da-kube-api-access-ckn2j\") pod \"7ff1e722-5ad1-448c-9339-a26551fe84da\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") "
Nov 03 07:45:48.378463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:48.378319    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-openshift-ca\") pod \"7ff1e722-5ad1-448c-9339-a26551fe84da\" (UID: \"7ff1e722-5ad1-448c-9339-a26551fe84da\") "
Nov 03 07:45:48.378726 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:48.378700    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-openshift-ca" pod "7ff1e722-5ad1-448c-9339-a26551fe84da" (UID: "7ff1e722-5ad1-448c-9339-a26551fe84da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:48.380581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:48.380558    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-xks-ca" pod "7ff1e722-5ad1-448c-9339-a26551fe84da" (UID: "7ff1e722-5ad1-448c-9339-a26551fe84da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:45:48.380581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:48.380562    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff1e722-5ad1-448c-9339-a26551fe84da-kube-api-access-ckn2j" pod "7ff1e722-5ad1-448c-9339-a26551fe84da" (UID: "7ff1e722-5ad1-448c-9339-a26551fe84da"). InnerVolumeSpecName "kube-api-access-ckn2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:48.479845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:48.479808    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:48.479845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:48.479841    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckn2j\" (UniqueName: \"kubernetes.io/projected/7ff1e722-5ad1-448c-9339-a26551fe84da-kube-api-access-ckn2j\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:48.479845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:48.479850    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff1e722-5ad1-448c-9339-a26551fe84da-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:49.132285 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.132253    4470 generic.go:356] "Generic (PLEG): container finished" podID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerID="92d25661d9056277ba97bfb36c8b9ef9ee2ee56eb850278fdbc53e3c2ac80c96" exitCode=0
Nov 03 07:45:49.132462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.132326    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" event={"ID":"8605b360-6f53-41b3-a1ab-295c6bbdd046","Type":"ContainerDied","Data":"92d25661d9056277ba97bfb36c8b9ef9ee2ee56eb850278fdbc53e3c2ac80c96"}
Nov 03 07:45:49.132462 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.132400    4470 scope.go:122] "RemoveContainer" containerID="c327cf187cec1f3259aca10154dc78afda76cd2bbd1110ee0e92519a1b2c21da"
Nov 03 07:45:49.138500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.138469    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm" event={"ID":"7ff1e722-5ad1-448c-9339-a26551fe84da","Type":"ContainerDied","Data":"67a3dcb11fbfebbc3167722e285fdee5590fcfd7baa0cad34c44ba34256facd6"}
Nov 03 07:45:49.138720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.138529    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67a3dcb11fbfebbc3167722e285fdee5590fcfd7baa0cad34c44ba34256facd6"
Nov 03 07:45:49.138720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.138598    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"
Nov 03 07:45:49.193920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.193894    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:45:49.287848 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.287810    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-certs\") pod \"8605b360-6f53-41b3-a1ab-295c6bbdd046\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") "
Nov 03 07:45:49.288376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.287900    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-envoy\") pod \"8605b360-6f53-41b3-a1ab-295c6bbdd046\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") "
Nov 03 07:45:49.288376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.287988    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-kube-api-access-tm8xf\" (UniqueName: \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-kube-api-access-tm8xf\") pod \"8605b360-6f53-41b3-a1ab-295c6bbdd046\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") "
Nov 03 07:45:49.288376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.288066    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-credential-socket\") pod \"8605b360-6f53-41b3-a1ab-295c6bbdd046\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") "
Nov 03 07:45:49.288376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.288121    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8605b360-6f53-41b3-a1ab-295c6bbdd046-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8605b360-6f53-41b3-a1ab-295c6bbdd046-istiod-ca-cert\") pod \"8605b360-6f53-41b3-a1ab-295c6bbdd046\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") "
Nov 03 07:45:49.288376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.288211    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-podinfo\") pod \"8605b360-6f53-41b3-a1ab-295c6bbdd046\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") "
Nov 03 07:45:49.288376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.288121    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-certs" pod "8605b360-6f53-41b3-a1ab-295c6bbdd046" (UID: "8605b360-6f53-41b3-a1ab-295c6bbdd046"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:49.288376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.288272    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-token\" (UniqueName: \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-token\") pod \"8605b360-6f53-41b3-a1ab-295c6bbdd046\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") "
Nov 03 07:45:49.288376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.288313    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-socket\") pod \"8605b360-6f53-41b3-a1ab-295c6bbdd046\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") "
Nov 03 07:45:49.288376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.288362    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-data\") pod \"8605b360-6f53-41b3-a1ab-295c6bbdd046\" (UID: \"8605b360-6f53-41b3-a1ab-295c6bbdd046\") "
Nov 03 07:45:49.288376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.288370    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-credential-socket" pod "8605b360-6f53-41b3-a1ab-295c6bbdd046" (UID: "8605b360-6f53-41b3-a1ab-295c6bbdd046"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:49.288828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.288493    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8605b360-6f53-41b3-a1ab-295c6bbdd046-istiod-ca-cert" pod "8605b360-6f53-41b3-a1ab-295c6bbdd046" (UID: "8605b360-6f53-41b3-a1ab-295c6bbdd046"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:49.288828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.288592    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-socket" pod "8605b360-6f53-41b3-a1ab-295c6bbdd046" (UID: "8605b360-6f53-41b3-a1ab-295c6bbdd046"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:49.288828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.288673    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-data" pod "8605b360-6f53-41b3-a1ab-295c6bbdd046" (UID: "8605b360-6f53-41b3-a1ab-295c6bbdd046"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:49.289156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.289106    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:49.289441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.289175    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:49.289441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.289196    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:49.289441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.289213    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8605b360-6f53-41b3-a1ab-295c6bbdd046-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:49.289441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.289226    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:49.290346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.290318    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-envoy" pod "8605b360-6f53-41b3-a1ab-295c6bbdd046" (UID: "8605b360-6f53-41b3-a1ab-295c6bbdd046"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:49.290498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.290477    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-podinfo" pod "8605b360-6f53-41b3-a1ab-295c6bbdd046" (UID: "8605b360-6f53-41b3-a1ab-295c6bbdd046"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:45:49.290564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.290546    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-token" pod "8605b360-6f53-41b3-a1ab-295c6bbdd046" (UID: "8605b360-6f53-41b3-a1ab-295c6bbdd046"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:49.290961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.290946    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-kube-api-access-tm8xf" pod "8605b360-6f53-41b3-a1ab-295c6bbdd046" (UID: "8605b360-6f53-41b3-a1ab-295c6bbdd046"). InnerVolumeSpecName "kube-api-access-tm8xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:49.390025 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.389991    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:49.390025 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.390021    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm8xf\" (UniqueName: \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-kube-api-access-tm8xf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:49.390025 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.390031    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:49.390303 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.390039    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8605b360-6f53-41b3-a1ab-295c6bbdd046-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:49.412197 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.412162    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"]
Nov 03 07:45:49.420391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:49.420361    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"]
Nov 03 07:45:50.146388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:50.146306    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"
Nov 03 07:45:50.146388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:50.146320    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm" event={"ID":"8605b360-6f53-41b3-a1ab-295c6bbdd046","Type":"ContainerDied","Data":"68dc2a0f7f8dbe17ef8fad93ab729324326b7adeaf89467238e51c13e82919b3"}
Nov 03 07:45:50.146388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:50.146379    4470 scope.go:122] "RemoveContainer" containerID="92d25661d9056277ba97bfb36c8b9ef9ee2ee56eb850278fdbc53e3c2ac80c96"
Nov 03 07:45:50.223357 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:50.223322    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"]
Nov 03 07:45:50.318601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:50.318569    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"]
Nov 03 07:45:50.325342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:50.325316    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm"]
Nov 03 07:45:50.334208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:50.334181    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"]
Nov 03 07:45:51.807037 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:51.807005    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" path="/var/lib/kubelet/pods/8605b360-6f53-41b3-a1ab-295c6bbdd046/volumes"
Nov 03 07:45:55.223872 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.223808    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2" podUID="dd115615-a8ce-4efa-bc4f-5d32c851601f" containerName="payload-processing" containerID="cri-o://82613b64e6724240fd2bd79b11fe2521a1fe54f43d6e7af4225b40c2686ac9c1" gracePeriod=25
Nov 03 07:45:55.335405 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.335343    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq" podUID="1fc99087-3fdd-4df7-9bf6-7624dc189249" containerName="payload-pre-processing" containerID="cri-o://7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60" gracePeriod=25
Nov 03 07:45:55.899450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.899416    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"]
Nov 03 07:45:55.899937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.899923    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy"
Nov 03 07:45:55.900020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.899938    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy"
Nov 03 07:45:55.900020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.899968    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7ff1e722-5ad1-448c-9339-a26551fe84da" containerName="revoke-keys"
Nov 03 07:45:55.900020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.899973    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff1e722-5ad1-448c-9339-a26551fe84da" containerName="revoke-keys"
Nov 03 07:45:55.900020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.899982    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy"
Nov 03 07:45:55.900020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.899987    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy"
Nov 03 07:45:55.900302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.900070    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy"
Nov 03 07:45:55.900302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.900083    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff1e722-5ad1-448c-9339-a26551fe84da" containerName="revoke-keys"
Nov 03 07:45:55.904743 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.904724    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:55.958680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.958645    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"]
Nov 03 07:45:55.959275 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.959260    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8605b360-6f53-41b3-a1ab-295c6bbdd046" containerName="istio-proxy"
Nov 03 07:45:55.962865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.962837    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:55.964771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:55.964745    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"]
Nov 03 07:45:56.006634 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.006606    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:45:56.006813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.006618    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:45:56.006813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.006635    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:45:56.011654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.011629    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:45:56.022874 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.022841    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"]
Nov 03 07:45:56.066611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.066579    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c41d424-54pxw\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:56.066771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.066628    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1208e6-wm4tf\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:56.066771 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.066744    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxc4\" (UniqueName: \"kubernetes.io/projected/28386bf2-f823-49d3-bf34-06145054f1bc-kube-api-access-nmxc4\") pod \"maas-api-revoke-sub-4c41d424-54pxw\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:56.066868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.066776    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1208e6-wm4tf\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:56.066901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.066879    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5x4j\" (UniqueName: \"kubernetes.io/projected/a34be0f8-6024-477b-864d-6eff1225d453-kube-api-access-z5x4j\") pod \"maas-api-revoke-sub-ab1208e6-wm4tf\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:56.066999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.066986    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c41d424-54pxw\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:56.168322 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.168240    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c41d424-54pxw\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:56.168322 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.168292    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1208e6-wm4tf\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:56.168524 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.168343    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxc4\" (UniqueName: \"kubernetes.io/projected/28386bf2-f823-49d3-bf34-06145054f1bc-kube-api-access-nmxc4\") pod \"maas-api-revoke-sub-4c41d424-54pxw\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:56.168524 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.168360    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1208e6-wm4tf\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:56.168524 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.168415    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5x4j\" (UniqueName: \"kubernetes.io/projected/a34be0f8-6024-477b-864d-6eff1225d453-kube-api-access-z5x4j\") pod \"maas-api-revoke-sub-ab1208e6-wm4tf\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:56.168524 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.168494    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c41d424-54pxw\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:56.169006 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.168983    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c41d424-54pxw\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:56.169288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.169078    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1208e6-wm4tf\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:56.170907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.170884    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c41d424-54pxw\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:56.171002 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.170965    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1208e6-wm4tf\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:56.232739 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.232709    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5x4j\" (UniqueName: \"kubernetes.io/projected/a34be0f8-6024-477b-864d-6eff1225d453-kube-api-access-z5x4j\") pod \"maas-api-revoke-sub-ab1208e6-wm4tf\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:56.233108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.232803    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxc4\" (UniqueName: \"kubernetes.io/projected/28386bf2-f823-49d3-bf34-06145054f1bc-kube-api-access-nmxc4\") pod \"maas-api-revoke-sub-4c41d424-54pxw\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:56.278053 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.278012    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:56.495962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.495884    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"]
Nov 03 07:45:56.518338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.518306    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:56.605584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.605550    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"]
Nov 03 07:45:56.658267 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.658239    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"]
Nov 03 07:45:56.659284 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:45:56.659260    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34be0f8_6024_477b_864d_6eff1225d453.slice/crio-8fa109270b64212ffbec7dcbdb5d49a4c82e2bfa51e0ce1cb698b8937d5ec086 WatchSource:0}: Error finding container 8fa109270b64212ffbec7dcbdb5d49a4c82e2bfa51e0ce1cb698b8937d5ec086: Status 404 returned error can't find the container with id 8fa109270b64212ffbec7dcbdb5d49a4c82e2bfa51e0ce1cb698b8937d5ec086
Nov 03 07:45:56.896874 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:56.896847    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"]
Nov 03 07:45:56.897787 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:45:56.897760    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28386bf2_f823_49d3_bf34_06145054f1bc.slice/crio-e1aebb7a2ae4857f5e71cad6dc89d8b1585361fa7b9f1a99646ed5758616c8cd WatchSource:0}: Error finding container e1aebb7a2ae4857f5e71cad6dc89d8b1585361fa7b9f1a99646ed5758616c8cd: Status 404 returned error can't find the container with id e1aebb7a2ae4857f5e71cad6dc89d8b1585361fa7b9f1a99646ed5758616c8cd
Nov 03 07:45:57.196339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:57.196303    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw" event={"ID":"28386bf2-f823-49d3-bf34-06145054f1bc","Type":"ContainerStarted","Data":"e1aebb7a2ae4857f5e71cad6dc89d8b1585361fa7b9f1a99646ed5758616c8cd"}
Nov 03 07:45:57.198282 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:57.198257    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf" event={"ID":"a34be0f8-6024-477b-864d-6eff1225d453","Type":"ContainerStarted","Data":"03b975d2c518f039c4dce91650083cd11f0b3019ff35a05b8fb45a3a2b1aecd5"}
Nov 03 07:45:57.198440 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:57.198303    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf" event={"ID":"a34be0f8-6024-477b-864d-6eff1225d453","Type":"ContainerStarted","Data":"8fa109270b64212ffbec7dcbdb5d49a4c82e2bfa51e0ce1cb698b8937d5ec086"}
Nov 03 07:45:57.320451 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:57.320396    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf" podStartSLOduration=2.320381882 podStartE2EDuration="2.320381882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:57.317688391 +0000 UTC m=+2809.650057384" watchObservedRunningTime="2026-11-03 07:45:57.320381882 +0000 UTC m=+2809.652750875"
Nov 03 07:45:58.164520 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.164469    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main" containerID="cri-o://f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0" gracePeriod=45
Nov 03 07:45:58.207686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.207646    4470 generic.go:356] "Generic (PLEG): container finished" podID="28386bf2-f823-49d3-bf34-06145054f1bc" containerID="47133ac6e334690d544058d2fa2c626ab42796667ea51999c69294aa77cb9717" exitCode=0
Nov 03 07:45:58.207884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.207722    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw" event={"ID":"28386bf2-f823-49d3-bf34-06145054f1bc","Type":"ContainerDied","Data":"47133ac6e334690d544058d2fa2c626ab42796667ea51999c69294aa77cb9717"}
Nov 03 07:45:58.209467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.209441    4470 generic.go:356] "Generic (PLEG): container finished" podID="a34be0f8-6024-477b-864d-6eff1225d453" containerID="03b975d2c518f039c4dce91650083cd11f0b3019ff35a05b8fb45a3a2b1aecd5" exitCode=0
Nov 03 07:45:58.209589 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.209532    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf" event={"ID":"a34be0f8-6024-477b-864d-6eff1225d453","Type":"ContainerDied","Data":"03b975d2c518f039c4dce91650083cd11f0b3019ff35a05b8fb45a3a2b1aecd5"}
Nov 03 07:45:58.460867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.460836    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:45:58.493425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.493394    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-model-cache\") pod \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") "
Nov 03 07:45:58.493425 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.493425    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-dshm\") pod \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") "
Nov 03 07:45:58.493679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.493466    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-home\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-home\") pod \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") "
Nov 03 07:45:58.493679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.493494    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tls-certs\" (UniqueName: \"kubernetes.io/secret/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tls-certs\") pod \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") "
Nov 03 07:45:58.493679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.493586    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-kube-api-access-68plg\" (UniqueName: \"kubernetes.io/projected/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-kube-api-access-68plg\") pod \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") "
Nov 03 07:45:58.493854 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.493686    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tmp-dir\") pod \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\" (UID: \"60fa78f7-82a6-4265-bafc-3f09f8cb7d39\") "
Nov 03 07:45:58.493854 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.493730    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-home" pod "60fa78f7-82a6-4265-bafc-3f09f8cb7d39" (UID: "60fa78f7-82a6-4265-bafc-3f09f8cb7d39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:58.493854 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.493741    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-model-cache" pod "60fa78f7-82a6-4265-bafc-3f09f8cb7d39" (UID: "60fa78f7-82a6-4265-bafc-3f09f8cb7d39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:58.494061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.494020    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tmp-dir" pod "60fa78f7-82a6-4265-bafc-3f09f8cb7d39" (UID: "60fa78f7-82a6-4265-bafc-3f09f8cb7d39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:58.494245 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.494223    4470 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tmp-dir\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:58.494341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.494250    4470 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-model-cache\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:58.494341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.494266    4470 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-home\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:58.495968 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.495947    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-dshm" pod "60fa78f7-82a6-4265-bafc-3f09f8cb7d39" (UID: "60fa78f7-82a6-4265-bafc-3f09f8cb7d39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:45:58.496075 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.496044    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-kube-api-access-68plg" pod "60fa78f7-82a6-4265-bafc-3f09f8cb7d39" (UID: "60fa78f7-82a6-4265-bafc-3f09f8cb7d39"). InnerVolumeSpecName "kube-api-access-68plg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:58.496355 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.496335    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tls-certs" pod "60fa78f7-82a6-4265-bafc-3f09f8cb7d39" (UID: "60fa78f7-82a6-4265-bafc-3f09f8cb7d39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:45:58.595751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.595719    4470 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-dshm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:58.595751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.595750    4470 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-tls-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:58.595949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.595763    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68plg\" (UniqueName: \"kubernetes.io/projected/60fa78f7-82a6-4265-bafc-3f09f8cb7d39-kube-api-access-68plg\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:58.931699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.931659    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq" podUID="1fc99087-3fdd-4df7-9bf6-7624dc189249" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:45:58.931699 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:58.931679    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2" podUID="dd115615-a8ce-4efa-bc4f-5d32c851601f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:45:59.218043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.217951    4470 generic.go:356] "Generic (PLEG): container finished" podID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerID="f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0" exitCode=0
Nov 03 07:45:59.218043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.217998    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"
Nov 03 07:45:59.218340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.218035    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" event={"ID":"60fa78f7-82a6-4265-bafc-3f09f8cb7d39","Type":"ContainerDied","Data":"f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0"}
Nov 03 07:45:59.218340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.218084    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s" event={"ID":"60fa78f7-82a6-4265-bafc-3f09f8cb7d39","Type":"ContainerDied","Data":"6ccd8d732fc5c0745a3509ec37b51f1a56506de7b696175e1b5f24c32f03c041"}
Nov 03 07:45:59.218340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.218105    4470 scope.go:122] "RemoveContainer" containerID="f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0"
Nov 03 07:45:59.230683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.230665    4470 scope.go:122] "RemoveContainer" containerID="f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0"
Nov 03 07:45:59.230983 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:45:59.230958    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0\": container with ID starting with f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0 not found: ID does not exist" containerID="f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0"
Nov 03 07:45:59.231074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.230995    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0"} err="failed to get container status \"f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0\": rpc error: code = NotFound desc = could not find container \"f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0\": container with ID starting with f90bb441e464fe7c467fbea42cf271972899ddb45387d4b6061ee075e28cceb0 not found: ID does not exist"
Nov 03 07:45:59.370971 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.370945    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:45:59.394558 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.394530    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"]
Nov 03 07:45:59.402437 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.402408    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:45:59.404638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.404613    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-xks-ca\") pod \"a34be0f8-6024-477b-864d-6eff1225d453\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") "
Nov 03 07:45:59.404766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.404672    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-openshift-ca\") pod \"a34be0f8-6024-477b-864d-6eff1225d453\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") "
Nov 03 07:45:59.404766 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.404747    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34be0f8-6024-477b-864d-6eff1225d453-kube-api-access-z5x4j\" (UniqueName: \"kubernetes.io/projected/a34be0f8-6024-477b-864d-6eff1225d453-kube-api-access-z5x4j\") pod \"a34be0f8-6024-477b-864d-6eff1225d453\" (UID: \"a34be0f8-6024-477b-864d-6eff1225d453\") "
Nov 03 07:45:59.405018 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.404992    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-openshift-ca" pod "a34be0f8-6024-477b-864d-6eff1225d453" (UID: "a34be0f8-6024-477b-864d-6eff1225d453"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:59.405295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.405277    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:59.406841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.406820    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-xks-ca" pod "a34be0f8-6024-477b-864d-6eff1225d453" (UID: "a34be0f8-6024-477b-864d-6eff1225d453"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:45:59.406937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.406871    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34be0f8-6024-477b-864d-6eff1225d453-kube-api-access-z5x4j" pod "a34be0f8-6024-477b-864d-6eff1225d453" (UID: "a34be0f8-6024-477b-864d-6eff1225d453"). InnerVolumeSpecName "kube-api-access-z5x4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:59.442078 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.442055    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-2-w3-1f2e7e-kserve-58d449f5f9-shc7s"]
Nov 03 07:45:59.505920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.505841    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-openshift-ca\") pod \"28386bf2-f823-49d3-bf34-06145054f1bc\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") "
Nov 03 07:45:59.505920 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.505878    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28386bf2-f823-49d3-bf34-06145054f1bc-kube-api-access-nmxc4\" (UniqueName: \"kubernetes.io/projected/28386bf2-f823-49d3-bf34-06145054f1bc-kube-api-access-nmxc4\") pod \"28386bf2-f823-49d3-bf34-06145054f1bc\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") "
Nov 03 07:45:59.506368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.505960    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-xks-ca\") pod \"28386bf2-f823-49d3-bf34-06145054f1bc\" (UID: \"28386bf2-f823-49d3-bf34-06145054f1bc\") "
Nov 03 07:45:59.506368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.506197    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-openshift-ca" pod "28386bf2-f823-49d3-bf34-06145054f1bc" (UID: "28386bf2-f823-49d3-bf34-06145054f1bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:45:59.506368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.506360    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34be0f8-6024-477b-864d-6eff1225d453-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:59.506478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.506374    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:59.506478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.506384    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5x4j\" (UniqueName: \"kubernetes.io/projected/a34be0f8-6024-477b-864d-6eff1225d453-kube-api-access-z5x4j\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:59.508277 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.508254    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-xks-ca" pod "28386bf2-f823-49d3-bf34-06145054f1bc" (UID: "28386bf2-f823-49d3-bf34-06145054f1bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:45:59.508337 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.508276    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28386bf2-f823-49d3-bf34-06145054f1bc-kube-api-access-nmxc4" pod "28386bf2-f823-49d3-bf34-06145054f1bc" (UID: "28386bf2-f823-49d3-bf34-06145054f1bc"). InnerVolumeSpecName "kube-api-access-nmxc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:45:59.607685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.607652    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28386bf2-f823-49d3-bf34-06145054f1bc-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:59.607685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.607681    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmxc4\" (UniqueName: \"kubernetes.io/projected/28386bf2-f823-49d3-bf34-06145054f1bc-kube-api-access-nmxc4\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:45:59.807040 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:45:59.806968    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" path="/var/lib/kubelet/pods/60fa78f7-82a6-4265-bafc-3f09f8cb7d39/volumes"
Nov 03 07:46:00.225239 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:00.225198    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw" event={"ID":"28386bf2-f823-49d3-bf34-06145054f1bc","Type":"ContainerDied","Data":"e1aebb7a2ae4857f5e71cad6dc89d8b1585361fa7b9f1a99646ed5758616c8cd"}
Nov 03 07:46:00.225452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:00.225260    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1aebb7a2ae4857f5e71cad6dc89d8b1585361fa7b9f1a99646ed5758616c8cd"
Nov 03 07:46:00.225452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:00.225211    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"
Nov 03 07:46:00.226648 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:00.226627    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"
Nov 03 07:46:00.226648 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:00.226636    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf" event={"ID":"a34be0f8-6024-477b-864d-6eff1225d453","Type":"ContainerDied","Data":"8fa109270b64212ffbec7dcbdb5d49a4c82e2bfa51e0ce1cb698b8937d5ec086"}
Nov 03 07:46:00.226797 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:00.226683    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fa109270b64212ffbec7dcbdb5d49a4c82e2bfa51e0ce1cb698b8937d5ec086"
Nov 03 07:46:01.496452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.496414    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v" podUID="3609292e-abcc-4a77-b03b-4f3d88ffa2e8" containerName="payload-processing" containerID="cri-o://577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b" gracePeriod=25
Nov 03 07:46:01.606458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.606416    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk" podUID="f27b5939-0090-4f1e-a339-472a5369058b" containerName="payload-pre-processing" containerID="cri-o://49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1" gracePeriod=25
Nov 03 07:46:01.765223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.765197    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:46:01.832389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.832358    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-plugins-config-volume\") pod \"3609292e-abcc-4a77-b03b-4f3d88ffa2e8\" (UID: \"3609292e-abcc-4a77-b03b-4f3d88ffa2e8\") "
Nov 03 07:46:01.832594 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.832583    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-kube-api-access-vl9mj\" (UniqueName: \"kubernetes.io/projected/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-kube-api-access-vl9mj\") pod \"3609292e-abcc-4a77-b03b-4f3d88ffa2e8\" (UID: \"3609292e-abcc-4a77-b03b-4f3d88ffa2e8\") "
Nov 03 07:46:01.833033 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.832979    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-plugins-config-volume" pod "3609292e-abcc-4a77-b03b-4f3d88ffa2e8" (UID: "3609292e-abcc-4a77-b03b-4f3d88ffa2e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:46:01.833250 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.833234    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:01.838379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.838302    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-kube-api-access-vl9mj" pod "3609292e-abcc-4a77-b03b-4f3d88ffa2e8" (UID: "3609292e-abcc-4a77-b03b-4f3d88ffa2e8"). InnerVolumeSpecName "kube-api-access-vl9mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:01.883368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.883341    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:46:01.934188 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.934061    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27b5939-0090-4f1e-a339-472a5369058b-kube-api-access-66d2h\" (UniqueName: \"kubernetes.io/projected/f27b5939-0090-4f1e-a339-472a5369058b-kube-api-access-66d2h\") pod \"f27b5939-0090-4f1e-a339-472a5369058b\" (UID: \"f27b5939-0090-4f1e-a339-472a5369058b\") "
Nov 03 07:46:01.934188 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.934114    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27b5939-0090-4f1e-a339-472a5369058b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27b5939-0090-4f1e-a339-472a5369058b-plugins-config-volume\") pod \"f27b5939-0090-4f1e-a339-472a5369058b\" (UID: \"f27b5939-0090-4f1e-a339-472a5369058b\") "
Nov 03 07:46:01.934514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.934490    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27b5939-0090-4f1e-a339-472a5369058b-plugins-config-volume" pod "f27b5939-0090-4f1e-a339-472a5369058b" (UID: "f27b5939-0090-4f1e-a339-472a5369058b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:46:01.934637 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.934623    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27b5939-0090-4f1e-a339-472a5369058b-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:01.934684 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.934640    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl9mj\" (UniqueName: \"kubernetes.io/projected/3609292e-abcc-4a77-b03b-4f3d88ffa2e8-kube-api-access-vl9mj\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:01.936364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:01.936340    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27b5939-0090-4f1e-a339-472a5369058b-kube-api-access-66d2h" pod "f27b5939-0090-4f1e-a339-472a5369058b" (UID: "f27b5939-0090-4f1e-a339-472a5369058b"). InnerVolumeSpecName "kube-api-access-66d2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:02.036579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.036535    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66d2h\" (UniqueName: \"kubernetes.io/projected/f27b5939-0090-4f1e-a339-472a5369058b-kube-api-access-66d2h\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:02.242594 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.242510    4470 generic.go:356] "Generic (PLEG): container finished" podID="f27b5939-0090-4f1e-a339-472a5369058b" containerID="49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1" exitCode=0
Nov 03 07:46:02.242594 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.242565    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"
Nov 03 07:46:02.242826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.242586    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk" event={"ID":"f27b5939-0090-4f1e-a339-472a5369058b","Type":"ContainerDied","Data":"49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1"}
Nov 03 07:46:02.242826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.242669    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk" event={"ID":"f27b5939-0090-4f1e-a339-472a5369058b","Type":"ContainerDied","Data":"2ab60e14d2aacce2199fb9c4c17d93de8c478703be6b36f1ff45cdb832f38973"}
Nov 03 07:46:02.242826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.242763    4470 scope.go:122] "RemoveContainer" containerID="49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1"
Nov 03 07:46:02.244489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.244454    4470 generic.go:356] "Generic (PLEG): container finished" podID="3609292e-abcc-4a77-b03b-4f3d88ffa2e8" containerID="577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b" exitCode=0
Nov 03 07:46:02.244598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.244516    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"
Nov 03 07:46:02.244646 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.244613    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v" event={"ID":"3609292e-abcc-4a77-b03b-4f3d88ffa2e8","Type":"ContainerDied","Data":"577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b"}
Nov 03 07:46:02.244682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.244649    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v" event={"ID":"3609292e-abcc-4a77-b03b-4f3d88ffa2e8","Type":"ContainerDied","Data":"5114abadc11b6e85224a1070c05d108fb5ef4764224c4e5a05b49362ddb3215b"}
Nov 03 07:46:02.252919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.252773    4470 scope.go:122] "RemoveContainer" containerID="49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1"
Nov 03 07:46:02.253279 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:46:02.253248    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1\": container with ID starting with 49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1 not found: ID does not exist" containerID="49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1"
Nov 03 07:46:02.253358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.253289    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1"} err="failed to get container status \"49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1\": rpc error: code = NotFound desc = could not find container \"49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1\": container with ID starting with 49306920b04bda989d4751a98a99439deb409e5c67b98b49d2a458092b7eb1c1 not found: ID does not exist"
Nov 03 07:46:02.253358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.253312    4470 scope.go:122] "RemoveContainer" containerID="577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b"
Nov 03 07:46:02.262688 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.262671    4470 scope.go:122] "RemoveContainer" containerID="577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b"
Nov 03 07:46:02.263013 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:46:02.262991    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b\": container with ID starting with 577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b not found: ID does not exist" containerID="577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b"
Nov 03 07:46:02.263072 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.263019    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b"} err="failed to get container status \"577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b\": rpc error: code = NotFound desc = could not find container \"577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b\": container with ID starting with 577d2bf8c83fc5c4cb161f65920f0227540598b816d56b51a092bed31023658b not found: ID does not exist"
Nov 03 07:46:02.414307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.414274    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"]
Nov 03 07:46:02.465233 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.465198    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk"]
Nov 03 07:46:02.579297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.579262    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"]
Nov 03 07:46:02.587089 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:02.587053    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v"]
Nov 03 07:46:03.806490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:03.806460    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3609292e-abcc-4a77-b03b-4f3d88ffa2e8" path="/var/lib/kubelet/pods/3609292e-abcc-4a77-b03b-4f3d88ffa2e8/volumes"
Nov 03 07:46:03.806884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:03.806852    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27b5939-0090-4f1e-a339-472a5369058b" path="/var/lib/kubelet/pods/f27b5939-0090-4f1e-a339-472a5369058b/volumes"
Nov 03 07:46:04.434968 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.434924    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main" containerID="cri-o://dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350" gracePeriod=45
Nov 03 07:46:04.448549 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.448516    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main" containerID="cri-o://e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4" gracePeriod=45
Nov 03 07:46:04.712414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.712391    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:46:04.746420 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.746394    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:46:04.759737 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:46:04.759709    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:46:04.767356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.767328    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-home\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-home\") pod \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") "
Nov 03 07:46:04.767504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.767390    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-model-cache\") pod \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") "
Nov 03 07:46:04.767504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.767443    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tmp-dir\") pod \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") "
Nov 03 07:46:04.767644 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.767609    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-home" pod "6e10c86c-6b75-4686-937b-55c11fbf9ae9" (UID: "6e10c86c-6b75-4686-937b-55c11fbf9ae9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:04.767644 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.767620    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tls-certs\") pod \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") "
Nov 03 07:46:04.767744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.767648    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-model-cache" pod "6e10c86c-6b75-4686-937b-55c11fbf9ae9" (UID: "6e10c86c-6b75-4686-937b-55c11fbf9ae9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:04.767744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.767661    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-dshm\") pod \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") "
Nov 03 07:46:04.767744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.767688    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e10c86c-6b75-4686-937b-55c11fbf9ae9-kube-api-access-dwhj6\" (UniqueName: \"kubernetes.io/projected/6e10c86c-6b75-4686-937b-55c11fbf9ae9-kube-api-access-dwhj6\") pod \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\" (UID: \"6e10c86c-6b75-4686-937b-55c11fbf9ae9\") "
Nov 03 07:46:04.767744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.767724    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tmp-dir" pod "6e10c86c-6b75-4686-937b-55c11fbf9ae9" (UID: "6e10c86c-6b75-4686-937b-55c11fbf9ae9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:04.768291 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.768191    4470 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-home\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:04.768291 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.768219    4470 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-model-cache\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:04.768291 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.768258    4470 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tmp-dir\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:04.769998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.769976    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tls-certs" pod "6e10c86c-6b75-4686-937b-55c11fbf9ae9" (UID: "6e10c86c-6b75-4686-937b-55c11fbf9ae9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:46:04.769998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.769985    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-dshm" pod "6e10c86c-6b75-4686-937b-55c11fbf9ae9" (UID: "6e10c86c-6b75-4686-937b-55c11fbf9ae9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:04.770195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.770173    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e10c86c-6b75-4686-937b-55c11fbf9ae9-kube-api-access-dwhj6" pod "6e10c86c-6b75-4686-937b-55c11fbf9ae9" (UID: "6e10c86c-6b75-4686-937b-55c11fbf9ae9"). InnerVolumeSpecName "kube-api-access-dwhj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:04.869475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.869446    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-model-cache\") pod \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") "
Nov 03 07:46:04.869926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.869506    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tmp-dir\") pod \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") "
Nov 03 07:46:04.869926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.869531    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-home\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-home\") pod \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") "
Nov 03 07:46:04.869926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.869646    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-dshm\") pod \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") "
Nov 03 07:46:04.869926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.869682    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tls-certs\") pod \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") "
Nov 03 07:46:04.869926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.869735    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-kube-api-access-rjqn9\" (UniqueName: \"kubernetes.io/projected/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-kube-api-access-rjqn9\") pod \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\" (UID: \"7ddd1670-b8a3-4687-8f43-2918ad60b7a2\") "
Nov 03 07:46:04.869926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.869782    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-model-cache" pod "7ddd1670-b8a3-4687-8f43-2918ad60b7a2" (UID: "7ddd1670-b8a3-4687-8f43-2918ad60b7a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:04.869926 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.869840    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tmp-dir" pod "7ddd1670-b8a3-4687-8f43-2918ad60b7a2" (UID: "7ddd1670-b8a3-4687-8f43-2918ad60b7a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:04.870371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.870199    4470 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-model-cache\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:04.870371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.870220    4470 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tmp-dir\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:04.870371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.870237    4470 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e10c86c-6b75-4686-937b-55c11fbf9ae9-tls-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:04.870371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.870251    4470 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e10c86c-6b75-4686-937b-55c11fbf9ae9-dshm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:04.870371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.870267    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhj6\" (UniqueName: \"kubernetes.io/projected/6e10c86c-6b75-4686-937b-55c11fbf9ae9-kube-api-access-dwhj6\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:04.870564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.870499    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-home" pod "7ddd1670-b8a3-4687-8f43-2918ad60b7a2" (UID: "7ddd1670-b8a3-4687-8f43-2918ad60b7a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:04.872458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.872432    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-dshm" pod "7ddd1670-b8a3-4687-8f43-2918ad60b7a2" (UID: "7ddd1670-b8a3-4687-8f43-2918ad60b7a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:04.872678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.872647    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-kube-api-access-rjqn9" pod "7ddd1670-b8a3-4687-8f43-2918ad60b7a2" (UID: "7ddd1670-b8a3-4687-8f43-2918ad60b7a2"). InnerVolumeSpecName "kube-api-access-rjqn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:04.872930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.872905    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tls-certs" pod "7ddd1670-b8a3-4687-8f43-2918ad60b7a2" (UID: "7ddd1670-b8a3-4687-8f43-2918ad60b7a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:46:04.971935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.971866    4470 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-home\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:04.971935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.971893    4470 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-dshm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:04.971935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.971902    4470 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-tls-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:04.971935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:04.971911    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjqn9\" (UniqueName: \"kubernetes.io/projected/7ddd1670-b8a3-4687-8f43-2918ad60b7a2-kube-api-access-rjqn9\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:05.269325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.269237    4470 generic.go:356] "Generic (PLEG): container finished" podID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerID="dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350" exitCode=0
Nov 03 07:46:05.269325 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.269286    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"
Nov 03 07:46:05.269553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.269324    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" event={"ID":"6e10c86c-6b75-4686-937b-55c11fbf9ae9","Type":"ContainerDied","Data":"dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350"}
Nov 03 07:46:05.269553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.269371    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp" event={"ID":"6e10c86c-6b75-4686-937b-55c11fbf9ae9","Type":"ContainerDied","Data":"77f141f08c6da7adfbf33e67743ff9d42526e7c3b5324836588b7971aa939066"}
Nov 03 07:46:05.269553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.269387    4470 scope.go:122] "RemoveContainer" containerID="dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350"
Nov 03 07:46:05.270989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.270966    4470 generic.go:356] "Generic (PLEG): container finished" podID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerID="e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4" exitCode=0
Nov 03 07:46:05.271081 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.271002    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" event={"ID":"7ddd1670-b8a3-4687-8f43-2918ad60b7a2","Type":"ContainerDied","Data":"e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4"}
Nov 03 07:46:05.271081 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.271053    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2" event={"ID":"7ddd1670-b8a3-4687-8f43-2918ad60b7a2","Type":"ContainerDied","Data":"841f110adcaf5342e0ae0975b25c3de7dde19085d6252d5fc1f9b37e62d9ea6c"}
Nov 03 07:46:05.271194 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.271090    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"
Nov 03 07:46:05.281483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.281466    4470 scope.go:122] "RemoveContainer" containerID="dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350"
Nov 03 07:46:05.281760 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:46:05.281743    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350\": container with ID starting with dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350 not found: ID does not exist" containerID="dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350"
Nov 03 07:46:05.281813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.281767    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350"} err="failed to get container status \"dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350\": rpc error: code = NotFound desc = could not find container \"dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350\": container with ID starting with dacb89370cba94dd764040c5034be9e87ecec970b8e1e6218d0d557d1755e350 not found: ID does not exist"
Nov 03 07:46:05.281813 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.281786    4470 scope.go:122] "RemoveContainer" containerID="e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4"
Nov 03 07:46:05.290125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.290108    4470 scope.go:122] "RemoveContainer" containerID="e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4"
Nov 03 07:46:05.290392 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:46:05.290373    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4\": container with ID starting with e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4 not found: ID does not exist" containerID="e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4"
Nov 03 07:46:05.290450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.290400    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4"} err="failed to get container status \"e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4\": rpc error: code = NotFound desc = could not find container \"e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4\": container with ID starting with e15edbd761b52cfc47095a5175417e32130a81256fa9ef8f6f36f3e84bb226c4 not found: ID does not exist"
Nov 03 07:46:05.441015 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.440986    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"]
Nov 03 07:46:05.450673 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.450646    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-w2-669d6b-kserve-6f78f6b654-62lvp"]
Nov 03 07:46:05.561822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.561749    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"]
Nov 03 07:46:05.569613 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.569587    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/premium-w2-669d6b-kserve-6f8bd456d7-q9hp2"]
Nov 03 07:46:05.806969 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.806939    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" path="/var/lib/kubelet/pods/6e10c86c-6b75-4686-937b-55c11fbf9ae9/volumes"
Nov 03 07:46:05.807399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:05.807385    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" path="/var/lib/kubelet/pods/7ddd1670-b8a3-4687-8f43-2918ad60b7a2/volumes"
Nov 03 07:46:08.056790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:08.056757    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"]
Nov 03 07:46:08.107534 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:08.107500    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d14e0ee-bmxpr"]
Nov 03 07:46:08.931929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:08.931888    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2" podUID="dd115615-a8ce-4efa-bc4f-5d32c851601f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:46:08.932168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:08.931898    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq" podUID="1fc99087-3fdd-4df7-9bf6-7624dc189249" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:46:09.806473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:09.806444    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1408b8a1-960c-425e-8457-0a528f1e7fe6" path="/var/lib/kubelet/pods/1408b8a1-960c-425e-8457-0a528f1e7fe6/volumes"
Nov 03 07:46:13.382923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.382886    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"]
Nov 03 07:46:13.383428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383414    4470 cpu_manager.go:414] "containerMap: removing container" podUID="28386bf2-f823-49d3-bf34-06145054f1bc" containerName="revoke-keys"
Nov 03 07:46:13.383477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383429    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28386bf2-f823-49d3-bf34-06145054f1bc" containerName="revoke-keys"
Nov 03 07:46:13.383477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383443    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main"
Nov 03 07:46:13.383477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383448    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main"
Nov 03 07:46:13.383477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383470    4470 cpu_manager.go:414] "containerMap: removing container" podUID="f27b5939-0090-4f1e-a339-472a5369058b" containerName="payload-pre-processing"
Nov 03 07:46:13.383477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383476    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27b5939-0090-4f1e-a339-472a5369058b" containerName="payload-pre-processing"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383489    4470 cpu_manager.go:414] "containerMap: removing container" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383493    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383511    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3609292e-abcc-4a77-b03b-4f3d88ffa2e8" containerName="payload-processing"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383516    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3609292e-abcc-4a77-b03b-4f3d88ffa2e8" containerName="payload-processing"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383521    4470 cpu_manager.go:414] "containerMap: removing container" podUID="a34be0f8-6024-477b-864d-6eff1225d453" containerName="revoke-keys"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383526    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a34be0f8-6024-477b-864d-6eff1225d453" containerName="revoke-keys"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383542    4470 cpu_manager.go:414] "containerMap: removing container" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383546    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383633    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e10c86c-6b75-4686-937b-55c11fbf9ae9" containerName="main"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383644    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="28386bf2-f823-49d3-bf34-06145054f1bc" containerName="revoke-keys"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383650    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ddd1670-b8a3-4687-8f43-2918ad60b7a2" containerName="main"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383657    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27b5939-0090-4f1e-a339-472a5369058b" containerName="payload-pre-processing"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383663    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="60fa78f7-82a6-4265-bafc-3f09f8cb7d39" containerName="main"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383671    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3609292e-abcc-4a77-b03b-4f3d88ffa2e8" containerName="payload-processing"
Nov 03 07:46:13.383736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.383676    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="a34be0f8-6024-477b-864d-6eff1225d453" containerName="revoke-keys"
Nov 03 07:46:13.388522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.388503    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:13.445831 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.445802    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"]
Nov 03 07:46:13.446423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.446342    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:46:13.446423 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.446348    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:46:13.446615 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.446428    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:46:13.470660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.470623    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jx6m\" (UniqueName: \"kubernetes.io/projected/ad16193f-39cf-46e7-8bed-53234babb0fc-kube-api-access-2jx6m\") pod \"maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:13.470861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.470773    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:13.470861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.470806    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:13.492987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.492951    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:46:13.571776 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.571740    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jx6m\" (UniqueName: \"kubernetes.io/projected/ad16193f-39cf-46e7-8bed-53234babb0fc-kube-api-access-2jx6m\") pod \"maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:13.571994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.571874    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:13.571994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.571902    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:13.572736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.572662    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:13.574529 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.574508    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:13.635730 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.635659    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jx6m\" (UniqueName: \"kubernetes.io/projected/ad16193f-39cf-46e7-8bed-53234babb0fc-kube-api-access-2jx6m\") pod \"maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:13.701823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:13.701788    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:14.078475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:14.078441    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"]
Nov 03 07:46:14.080012 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:46:14.079988    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad16193f_39cf_46e7_8bed_53234babb0fc.slice/crio-b07cd42218cb30cfb08f92732ef848b97fe6bdf0de357faaf52ca6be704a9609 WatchSource:0}: Error finding container b07cd42218cb30cfb08f92732ef848b97fe6bdf0de357faaf52ca6be704a9609: Status 404 returned error can't find the container with id b07cd42218cb30cfb08f92732ef848b97fe6bdf0de357faaf52ca6be704a9609
Nov 03 07:46:14.347471 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:14.347377    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28" event={"ID":"ad16193f-39cf-46e7-8bed-53234babb0fc","Type":"ContainerStarted","Data":"b07cd42218cb30cfb08f92732ef848b97fe6bdf0de357faaf52ca6be704a9609"}
Nov 03 07:46:14.617660 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:14.617575    4470 scope.go:122] "RemoveContainer" containerID="f55938319a1482be5093c6b40ffa7fa2a000e03af67fd653a40518b7ede3a77e"
Nov 03 07:46:14.627740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:14.627720    4470 scope.go:122] "RemoveContainer" containerID="a0b905d331b6f0ac173c904c8a6d34dbcc52281b09e30a5c42d4437c28883031"
Nov 03 07:46:14.637822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:14.637800    4470 scope.go:122] "RemoveContainer" containerID="48ead51f5b5e1ea44984659793fb8d036360647ecbe92cccb41f84ebae648aee"
Nov 03 07:46:15.358287 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.358257    4470 generic.go:356] "Generic (PLEG): container finished" podID="dd115615-a8ce-4efa-bc4f-5d32c851601f" containerID="82613b64e6724240fd2bd79b11fe2521a1fe54f43d6e7af4225b40c2686ac9c1" exitCode=0
Nov 03 07:46:15.358490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.358316    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2" event={"ID":"dd115615-a8ce-4efa-bc4f-5d32c851601f","Type":"ContainerDied","Data":"82613b64e6724240fd2bd79b11fe2521a1fe54f43d6e7af4225b40c2686ac9c1"}
Nov 03 07:46:15.360163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.360108    4470 generic.go:356] "Generic (PLEG): container finished" podID="ad16193f-39cf-46e7-8bed-53234babb0fc" containerID="2a6b5d34ea5111c031e22dc4269d3f9e349db00f58dd4e409874cf1dc2919a32" exitCode=0
Nov 03 07:46:15.360298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.360163    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28" event={"ID":"ad16193f-39cf-46e7-8bed-53234babb0fc","Type":"ContainerDied","Data":"2a6b5d34ea5111c031e22dc4269d3f9e349db00f58dd4e409874cf1dc2919a32"}
Nov 03 07:46:15.494013 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.493990    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:46:15.596027 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.596002    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd115615-a8ce-4efa-bc4f-5d32c851601f-kube-api-access-7p4c6\" (UniqueName: \"kubernetes.io/projected/dd115615-a8ce-4efa-bc4f-5d32c851601f-kube-api-access-7p4c6\") pod \"dd115615-a8ce-4efa-bc4f-5d32c851601f\" (UID: \"dd115615-a8ce-4efa-bc4f-5d32c851601f\") "
Nov 03 07:46:15.596147 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.596093    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd115615-a8ce-4efa-bc4f-5d32c851601f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd115615-a8ce-4efa-bc4f-5d32c851601f-plugins-config-volume\") pod \"dd115615-a8ce-4efa-bc4f-5d32c851601f\" (UID: \"dd115615-a8ce-4efa-bc4f-5d32c851601f\") "
Nov 03 07:46:15.596492 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.596446    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd115615-a8ce-4efa-bc4f-5d32c851601f-plugins-config-volume" pod "dd115615-a8ce-4efa-bc4f-5d32c851601f" (UID: "dd115615-a8ce-4efa-bc4f-5d32c851601f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:46:15.598319 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.598297    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd115615-a8ce-4efa-bc4f-5d32c851601f-kube-api-access-7p4c6" pod "dd115615-a8ce-4efa-bc4f-5d32c851601f" (UID: "dd115615-a8ce-4efa-bc4f-5d32c851601f"). InnerVolumeSpecName "kube-api-access-7p4c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:15.615222 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.615197    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:46:15.697366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.697317    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc99087-3fdd-4df7-9bf6-7624dc189249-kube-api-access-n9n6j\" (UniqueName: \"kubernetes.io/projected/1fc99087-3fdd-4df7-9bf6-7624dc189249-kube-api-access-n9n6j\") pod \"1fc99087-3fdd-4df7-9bf6-7624dc189249\" (UID: \"1fc99087-3fdd-4df7-9bf6-7624dc189249\") "
Nov 03 07:46:15.697825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.697392    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc99087-3fdd-4df7-9bf6-7624dc189249-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc99087-3fdd-4df7-9bf6-7624dc189249-plugins-config-volume\") pod \"1fc99087-3fdd-4df7-9bf6-7624dc189249\" (UID: \"1fc99087-3fdd-4df7-9bf6-7624dc189249\") "
Nov 03 07:46:15.697825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.697721    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc99087-3fdd-4df7-9bf6-7624dc189249-plugins-config-volume" pod "1fc99087-3fdd-4df7-9bf6-7624dc189249" (UID: "1fc99087-3fdd-4df7-9bf6-7624dc189249"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:46:15.697974 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.697952    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd115615-a8ce-4efa-bc4f-5d32c851601f-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:15.698034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.697980    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc99087-3fdd-4df7-9bf6-7624dc189249-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:15.698034 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.697996    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p4c6\" (UniqueName: \"kubernetes.io/projected/dd115615-a8ce-4efa-bc4f-5d32c851601f-kube-api-access-7p4c6\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:15.699769 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.699748    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc99087-3fdd-4df7-9bf6-7624dc189249-kube-api-access-n9n6j" pod "1fc99087-3fdd-4df7-9bf6-7624dc189249" (UID: "1fc99087-3fdd-4df7-9bf6-7624dc189249"). InnerVolumeSpecName "kube-api-access-n9n6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:15.798941 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:15.798859    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9n6j\" (UniqueName: \"kubernetes.io/projected/1fc99087-3fdd-4df7-9bf6-7624dc189249-kube-api-access-n9n6j\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:16.367042 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.367012    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"
Nov 03 07:46:16.367298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.367042    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2" event={"ID":"dd115615-a8ce-4efa-bc4f-5d32c851601f","Type":"ContainerDied","Data":"363f94c76ed0193fbe6f4a6864169719b518b796af5e7d17deb3ac89a9af68da"}
Nov 03 07:46:16.367298 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.367108    4470 scope.go:122] "RemoveContainer" containerID="82613b64e6724240fd2bd79b11fe2521a1fe54f43d6e7af4225b40c2686ac9c1"
Nov 03 07:46:16.368514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.368491    4470 generic.go:356] "Generic (PLEG): container finished" podID="1fc99087-3fdd-4df7-9bf6-7624dc189249" containerID="7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60" exitCode=0
Nov 03 07:46:16.368619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.368524    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"
Nov 03 07:46:16.368619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.368570    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq" event={"ID":"1fc99087-3fdd-4df7-9bf6-7624dc189249","Type":"ContainerDied","Data":"7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60"}
Nov 03 07:46:16.368716 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.368635    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq" event={"ID":"1fc99087-3fdd-4df7-9bf6-7624dc189249","Type":"ContainerDied","Data":"a5b02234bdf1478ff42f4446debe421b80514bb6d02de33d776995e42c4638fb"}
Nov 03 07:46:16.377172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.377154    4470 scope.go:122] "RemoveContainer" containerID="7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60"
Nov 03 07:46:16.386453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.386435    4470 scope.go:122] "RemoveContainer" containerID="7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60"
Nov 03 07:46:16.386750 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:46:16.386728    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60\": container with ID starting with 7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60 not found: ID does not exist" containerID="7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60"
Nov 03 07:46:16.386829 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.386770    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60"} err="failed to get container status \"7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60\": rpc error: code = NotFound desc = could not find container \"7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60\": container with ID starting with 7e1005ebf6b3eb47a25f075d033ce633e4c089b5578ab334c7bdeed855ccba60 not found: ID does not exist"
Nov 03 07:46:16.508265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.508242    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:16.546356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.540358    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"]
Nov 03 07:46:16.588939 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.588910    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2"]
Nov 03 07:46:16.607269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.607244    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-openshift-ca\") pod \"ad16193f-39cf-46e7-8bed-53234babb0fc\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") "
Nov 03 07:46:16.607427 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.607367    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad16193f-39cf-46e7-8bed-53234babb0fc-kube-api-access-2jx6m\" (UniqueName: \"kubernetes.io/projected/ad16193f-39cf-46e7-8bed-53234babb0fc-kube-api-access-2jx6m\") pod \"ad16193f-39cf-46e7-8bed-53234babb0fc\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") "
Nov 03 07:46:16.607427 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.607413    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-xks-ca\") pod \"ad16193f-39cf-46e7-8bed-53234babb0fc\" (UID: \"ad16193f-39cf-46e7-8bed-53234babb0fc\") "
Nov 03 07:46:16.607612 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.607583    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-openshift-ca" pod "ad16193f-39cf-46e7-8bed-53234babb0fc" (UID: "ad16193f-39cf-46e7-8bed-53234babb0fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:46:16.607829 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.607811    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:16.609601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.609577    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-xks-ca" pod "ad16193f-39cf-46e7-8bed-53234babb0fc" (UID: "ad16193f-39cf-46e7-8bed-53234babb0fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:46:16.609601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.609596    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad16193f-39cf-46e7-8bed-53234babb0fc-kube-api-access-2jx6m" pod "ad16193f-39cf-46e7-8bed-53234babb0fc" (UID: "ad16193f-39cf-46e7-8bed-53234babb0fc"). InnerVolumeSpecName "kube-api-access-2jx6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:16.661950 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.661877    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"]
Nov 03 07:46:16.669326 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.669302    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq"]
Nov 03 07:46:16.709201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.709170    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jx6m\" (UniqueName: \"kubernetes.io/projected/ad16193f-39cf-46e7-8bed-53234babb0fc-kube-api-access-2jx6m\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:16.709201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:16.709197    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad16193f-39cf-46e7-8bed-53234babb0fc-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:17.377408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:17.377376    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"
Nov 03 07:46:17.377408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:17.377385    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28" event={"ID":"ad16193f-39cf-46e7-8bed-53234babb0fc","Type":"ContainerDied","Data":"b07cd42218cb30cfb08f92732ef848b97fe6bdf0de357faaf52ca6be704a9609"}
Nov 03 07:46:17.377648 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:17.377433    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b07cd42218cb30cfb08f92732ef848b97fe6bdf0de357faaf52ca6be704a9609"
Nov 03 07:46:17.807079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:17.806989    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc99087-3fdd-4df7-9bf6-7624dc189249" path="/var/lib/kubelet/pods/1fc99087-3fdd-4df7-9bf6-7624dc189249/volumes"
Nov 03 07:46:17.807630 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:17.807604    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd115615-a8ce-4efa-bc4f-5d32c851601f" path="/var/lib/kubelet/pods/dd115615-a8ce-4efa-bc4f-5d32c851601f/volumes"
Nov 03 07:46:26.603845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.603812    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"]
Nov 03 07:46:26.604434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.604414    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ad16193f-39cf-46e7-8bed-53234babb0fc" containerName="revoke-keys"
Nov 03 07:46:26.604491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.604436    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad16193f-39cf-46e7-8bed-53234babb0fc" containerName="revoke-keys"
Nov 03 07:46:26.604491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.604453    4470 cpu_manager.go:414] "containerMap: removing container" podUID="1fc99087-3fdd-4df7-9bf6-7624dc189249" containerName="payload-pre-processing"
Nov 03 07:46:26.604491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.604458    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc99087-3fdd-4df7-9bf6-7624dc189249" containerName="payload-pre-processing"
Nov 03 07:46:26.604491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.604470    4470 cpu_manager.go:414] "containerMap: removing container" podUID="dd115615-a8ce-4efa-bc4f-5d32c851601f" containerName="payload-processing"
Nov 03 07:46:26.604491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.604475    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd115615-a8ce-4efa-bc4f-5d32c851601f" containerName="payload-processing"
Nov 03 07:46:26.604658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.604583    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad16193f-39cf-46e7-8bed-53234babb0fc" containerName="revoke-keys"
Nov 03 07:46:26.604658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.604593    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc99087-3fdd-4df7-9bf6-7624dc189249" containerName="payload-pre-processing"
Nov 03 07:46:26.604658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.604601    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd115615-a8ce-4efa-bc4f-5d32c851601f" containerName="payload-processing"
Nov 03 07:46:26.609577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.609554    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:26.621929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.621903    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h56x5\" (UniqueName: \"kubernetes.io/projected/db17c026-3654-4351-b8d1-8f0d505626bd-kube-api-access-h56x5\") pod \"maas-api-revoke-sub-af13e332-2zxmv\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:26.622054 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.622023    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af13e332-2zxmv\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:26.622187 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.622159    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af13e332-2zxmv\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:26.664826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.664788    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"]
Nov 03 07:46:26.669381 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.669347    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"]
Nov 03 07:46:26.669557 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.669494    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:26.710006 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.709969    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:46:26.710207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.710046    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:46:26.713546 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.713521    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:46:26.713641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.713553    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:46:26.724896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.723774    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h56x5\" (UniqueName: \"kubernetes.io/projected/db17c026-3654-4351-b8d1-8f0d505626bd-kube-api-access-h56x5\") pod \"maas-api-revoke-sub-af13e332-2zxmv\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:26.724896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.723855    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fba7b73-zr6v7\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:26.724896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.723919    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af13e332-2zxmv\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:26.724896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.723937    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fhpc\" (UniqueName: \"kubernetes.io/projected/745e4642-5846-4cbb-bafc-9c2deb9f51bf-kube-api-access-8fhpc\") pod \"maas-api-revoke-sub-9fba7b73-zr6v7\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:26.724896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.723974    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fba7b73-zr6v7\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:26.724896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.724002    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af13e332-2zxmv\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:26.724896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.724850    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af13e332-2zxmv\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:26.738153 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.733196    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"]
Nov 03 07:46:26.742192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.742125    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af13e332-2zxmv\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:26.790046 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.790018    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h56x5\" (UniqueName: \"kubernetes.io/projected/db17c026-3654-4351-b8d1-8f0d505626bd-kube-api-access-h56x5\") pod \"maas-api-revoke-sub-af13e332-2zxmv\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:26.824818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.824789    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fba7b73-zr6v7\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:26.824977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.824910    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fba7b73-zr6v7\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:26.824977 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.824960    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fhpc\" (UniqueName: \"kubernetes.io/projected/745e4642-5846-4cbb-bafc-9c2deb9f51bf-kube-api-access-8fhpc\") pod \"maas-api-revoke-sub-9fba7b73-zr6v7\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:26.825488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.825467    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fba7b73-zr6v7\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:26.827426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.827402    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fba7b73-zr6v7\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:26.889021 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.888947    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fhpc\" (UniqueName: \"kubernetes.io/projected/745e4642-5846-4cbb-bafc-9c2deb9f51bf-kube-api-access-8fhpc\") pod \"maas-api-revoke-sub-9fba7b73-zr6v7\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:26.926265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.926236    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:26.990985 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:26.990917    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:27.304020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:27.303852    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"]
Nov 03 07:46:27.306729 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:46:27.306706    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb17c026_3654_4351_b8d1_8f0d505626bd.slice/crio-187f7a2f54cc823cf9d5fe06458c3511caf16a0bcd242f5ef9b0a334b0672bba WatchSource:0}: Error finding container 187f7a2f54cc823cf9d5fe06458c3511caf16a0bcd242f5ef9b0a334b0672bba: Status 404 returned error can't find the container with id 187f7a2f54cc823cf9d5fe06458c3511caf16a0bcd242f5ef9b0a334b0672bba
Nov 03 07:46:27.404071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:27.404049    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"]
Nov 03 07:46:27.405651 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:46:27.405625    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745e4642_5846_4cbb_bafc_9c2deb9f51bf.slice/crio-281220e0cac44f2f169f4312a8ab496fbf933de0a513f6ef178dcf3e26ed444a WatchSource:0}: Error finding container 281220e0cac44f2f169f4312a8ab496fbf933de0a513f6ef178dcf3e26ed444a: Status 404 returned error can't find the container with id 281220e0cac44f2f169f4312a8ab496fbf933de0a513f6ef178dcf3e26ed444a
Nov 03 07:46:27.438666 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:27.438630    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv" event={"ID":"db17c026-3654-4351-b8d1-8f0d505626bd","Type":"ContainerStarted","Data":"187f7a2f54cc823cf9d5fe06458c3511caf16a0bcd242f5ef9b0a334b0672bba"}
Nov 03 07:46:27.439827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:27.439804    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7" event={"ID":"745e4642-5846-4cbb-bafc-9c2deb9f51bf","Type":"ContainerStarted","Data":"281220e0cac44f2f169f4312a8ab496fbf933de0a513f6ef178dcf3e26ed444a"}
Nov 03 07:46:28.447282 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:28.447186    4470 generic.go:356] "Generic (PLEG): container finished" podID="db17c026-3654-4351-b8d1-8f0d505626bd" containerID="4e14e5711f185cde54ae64e9f6a93c2e638606e952f000d47666f6375905200e" exitCode=0
Nov 03 07:46:28.447282 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:28.447268    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv" event={"ID":"db17c026-3654-4351-b8d1-8f0d505626bd","Type":"ContainerDied","Data":"4e14e5711f185cde54ae64e9f6a93c2e638606e952f000d47666f6375905200e"}
Nov 03 07:46:28.449001 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:28.448973    4470 generic.go:356] "Generic (PLEG): container finished" podID="745e4642-5846-4cbb-bafc-9c2deb9f51bf" containerID="e0eab36e389a0d46f5bc0ce10ba34300b7794eee2cb27e1fa00814b2b25ca79b" exitCode=0
Nov 03 07:46:28.449169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:28.449056    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7" event={"ID":"745e4642-5846-4cbb-bafc-9c2deb9f51bf","Type":"ContainerDied","Data":"e0eab36e389a0d46f5bc0ce10ba34300b7794eee2cb27e1fa00814b2b25ca79b"}
Nov 03 07:46:29.597841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.597811    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:29.630673 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.630651    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:29.650971 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.650943    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-openshift-ca\") pod \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") "
Nov 03 07:46:29.651163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.650978    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745e4642-5846-4cbb-bafc-9c2deb9f51bf-kube-api-access-8fhpc\" (UniqueName: \"kubernetes.io/projected/745e4642-5846-4cbb-bafc-9c2deb9f51bf-kube-api-access-8fhpc\") pod \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") "
Nov 03 07:46:29.651163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.651003    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-xks-ca\") pod \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\" (UID: \"745e4642-5846-4cbb-bafc-9c2deb9f51bf\") "
Nov 03 07:46:29.651452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.651168    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-xks-ca\") pod \"db17c026-3654-4351-b8d1-8f0d505626bd\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") "
Nov 03 07:46:29.651452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.651223    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-openshift-ca\") pod \"db17c026-3654-4351-b8d1-8f0d505626bd\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") "
Nov 03 07:46:29.651452 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.651385    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-openshift-ca" pod "745e4642-5846-4cbb-bafc-9c2deb9f51bf" (UID: "745e4642-5846-4cbb-bafc-9c2deb9f51bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:46:29.651659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.651558    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db17c026-3654-4351-b8d1-8f0d505626bd-kube-api-access-h56x5\" (UniqueName: \"kubernetes.io/projected/db17c026-3654-4351-b8d1-8f0d505626bd-kube-api-access-h56x5\") pod \"db17c026-3654-4351-b8d1-8f0d505626bd\" (UID: \"db17c026-3654-4351-b8d1-8f0d505626bd\") "
Nov 03 07:46:29.651659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.651599    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-openshift-ca" pod "db17c026-3654-4351-b8d1-8f0d505626bd" (UID: "db17c026-3654-4351-b8d1-8f0d505626bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:46:29.652305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.652241    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:29.652305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.652270    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:29.653449 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.653425    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-xks-ca" pod "745e4642-5846-4cbb-bafc-9c2deb9f51bf" (UID: "745e4642-5846-4cbb-bafc-9c2deb9f51bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:46:29.653449 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.653433    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745e4642-5846-4cbb-bafc-9c2deb9f51bf-kube-api-access-8fhpc" pod "745e4642-5846-4cbb-bafc-9c2deb9f51bf" (UID: "745e4642-5846-4cbb-bafc-9c2deb9f51bf"). InnerVolumeSpecName "kube-api-access-8fhpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:29.653605 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.653472    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-xks-ca" pod "db17c026-3654-4351-b8d1-8f0d505626bd" (UID: "db17c026-3654-4351-b8d1-8f0d505626bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:46:29.653778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.653758    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db17c026-3654-4351-b8d1-8f0d505626bd-kube-api-access-h56x5" pod "db17c026-3654-4351-b8d1-8f0d505626bd" (UID: "db17c026-3654-4351-b8d1-8f0d505626bd"). InnerVolumeSpecName "kube-api-access-h56x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:29.753956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.753871    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h56x5\" (UniqueName: \"kubernetes.io/projected/db17c026-3654-4351-b8d1-8f0d505626bd-kube-api-access-h56x5\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:29.753956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.753903    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fhpc\" (UniqueName: \"kubernetes.io/projected/745e4642-5846-4cbb-bafc-9c2deb9f51bf-kube-api-access-8fhpc\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:29.753956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.753917    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745e4642-5846-4cbb-bafc-9c2deb9f51bf-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:29.753956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:29.753930    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db17c026-3654-4351-b8d1-8f0d505626bd-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:30.462602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:30.462565    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"
Nov 03 07:46:30.462602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:30.462589    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv" event={"ID":"db17c026-3654-4351-b8d1-8f0d505626bd","Type":"ContainerDied","Data":"187f7a2f54cc823cf9d5fe06458c3511caf16a0bcd242f5ef9b0a334b0672bba"}
Nov 03 07:46:30.462897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:30.462664    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="187f7a2f54cc823cf9d5fe06458c3511caf16a0bcd242f5ef9b0a334b0672bba"
Nov 03 07:46:30.464372 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:30.464348    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7" event={"ID":"745e4642-5846-4cbb-bafc-9c2deb9f51bf","Type":"ContainerDied","Data":"281220e0cac44f2f169f4312a8ab496fbf933de0a513f6ef178dcf3e26ed444a"}
Nov 03 07:46:30.464490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:30.464392    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="281220e0cac44f2f169f4312a8ab496fbf933de0a513f6ef178dcf3e26ed444a"
Nov 03 07:46:30.464490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:30.464441    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"
Nov 03 07:46:34.760305 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:46:34.760273    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:46:39.084001 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:39.083967    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"]
Nov 03 07:46:39.134362 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:39.134332    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1f2e7e-zcdwr"]
Nov 03 07:46:39.805436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:39.805402    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8ceb69-165f-4754-9df9-4da9496cc82e" path="/var/lib/kubelet/pods/aa8ceb69-165f-4754-9df9-4da9496cc82e/volumes"
Nov 03 07:46:40.401343 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:40.401309    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"]
Nov 03 07:46:40.452353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:40.452314    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e8bf4-bwp28"]
Nov 03 07:46:41.805465 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:41.805431    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad16193f-39cf-46e7-8bed-53234babb0fc" path="/var/lib/kubelet/pods/ad16193f-39cf-46e7-8bed-53234babb0fc/volumes"
Nov 03 07:46:43.060433 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:43.060403    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"]
Nov 03 07:46:43.111519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:43.111481    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"]
Nov 03 07:46:43.113953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:43.113933    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6686df-xrdzj"]
Nov 03 07:46:43.115961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:43.115940    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cceb04-plw96"]
Nov 03 07:46:43.806394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:43.806357    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1629fa1b-2823-4b28-8c15-110b8487c13f" path="/var/lib/kubelet/pods/1629fa1b-2823-4b28-8c15-110b8487c13f/volumes"
Nov 03 07:46:43.806791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:43.806779    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a" path="/var/lib/kubelet/pods/3ba9ecb3-32ef-4617-a1f7-7004d4a43d1a/volumes"
Nov 03 07:46:46.591850 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.591816    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"]
Nov 03 07:46:46.592504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.592479    4470 cpu_manager.go:414] "containerMap: removing container" podUID="db17c026-3654-4351-b8d1-8f0d505626bd" containerName="revoke-keys"
Nov 03 07:46:46.592504 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.592500    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db17c026-3654-4351-b8d1-8f0d505626bd" containerName="revoke-keys"
Nov 03 07:46:46.592663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.592513    4470 cpu_manager.go:414] "containerMap: removing container" podUID="745e4642-5846-4cbb-bafc-9c2deb9f51bf" containerName="revoke-keys"
Nov 03 07:46:46.592663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.592518    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745e4642-5846-4cbb-bafc-9c2deb9f51bf" containerName="revoke-keys"
Nov 03 07:46:46.592663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.592591    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="db17c026-3654-4351-b8d1-8f0d505626bd" containerName="revoke-keys"
Nov 03 07:46:46.592663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.592599    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="745e4642-5846-4cbb-bafc-9c2deb9f51bf" containerName="revoke-keys"
Nov 03 07:46:46.597107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.597090    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:46.654142 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.654112    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"]
Nov 03 07:46:46.701658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.701618    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:46:46.701658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.701659    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:46:46.701885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.701689    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:46:46.701885 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.701807    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:46:46.763378 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.763340    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxxd9\" (UniqueName: \"kubernetes.io/projected/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-kube-api-access-vxxd9\") pod \"maas-api-revoke-sub-ebbcd033-x5jtg\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:46.763579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.763393    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebbcd033-x5jtg\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:46.763579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.763532    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebbcd033-x5jtg\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:46.864824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.864736    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxd9\" (UniqueName: \"kubernetes.io/projected/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-kube-api-access-vxxd9\") pod \"maas-api-revoke-sub-ebbcd033-x5jtg\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:46.864824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.864787    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebbcd033-x5jtg\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:46.865086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.864853    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebbcd033-x5jtg\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:46.865683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.865664    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebbcd033-x5jtg\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:46.867565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.867542    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebbcd033-x5jtg\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:46.930063 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:46.930036    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxd9\" (UniqueName: \"kubernetes.io/projected/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-kube-api-access-vxxd9\") pod \"maas-api-revoke-sub-ebbcd033-x5jtg\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:47.210049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:47.209966    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:47.592817 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:47.592794    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"]
Nov 03 07:46:47.593625 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:46:47.593595    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f44b1f_4fc6_4f14_9e34_6f1230d5076f.slice/crio-193d609e3ee8cd63b7178dff6f7faa4ce86c608c1778ceec0c76060696729e93 WatchSource:0}: Error finding container 193d609e3ee8cd63b7178dff6f7faa4ce86c608c1778ceec0c76060696729e93: Status 404 returned error can't find the container with id 193d609e3ee8cd63b7178dff6f7faa4ce86c608c1778ceec0c76060696729e93
Nov 03 07:46:47.805914 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:47.805883    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"]
Nov 03 07:46:47.806108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:47.806074    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy" containerID="cri-o://d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c" gracePeriod=30
Nov 03 07:46:48.578743 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:48.578712    4470 generic.go:356] "Generic (PLEG): container finished" podID="19f44b1f-4fc6-4f14-9e34-6f1230d5076f" containerID="c223c845d3ad4b664e465c367dd0448d773d26ba1b3b7ac16608c3d825e0a681" exitCode=0
Nov 03 07:46:48.578934 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:48.578804    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg" event={"ID":"19f44b1f-4fc6-4f14-9e34-6f1230d5076f","Type":"ContainerDied","Data":"c223c845d3ad4b664e465c367dd0448d773d26ba1b3b7ac16608c3d825e0a681"}
Nov 03 07:46:48.578934 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:48.578856    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg" event={"ID":"19f44b1f-4fc6-4f14-9e34-6f1230d5076f","Type":"ContainerStarted","Data":"193d609e3ee8cd63b7178dff6f7faa4ce86c608c1778ceec0c76060696729e93"}
Nov 03 07:46:49.732712 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:49.732685    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:49.900801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:49.900727    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-xks-ca\") pod \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") "
Nov 03 07:46:49.900801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:49.900793    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-kube-api-access-vxxd9\" (UniqueName: \"kubernetes.io/projected/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-kube-api-access-vxxd9\") pod \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") "
Nov 03 07:46:49.900999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:49.900921    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-openshift-ca\") pod \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\" (UID: \"19f44b1f-4fc6-4f14-9e34-6f1230d5076f\") "
Nov 03 07:46:49.901333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:49.901300    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-openshift-ca" pod "19f44b1f-4fc6-4f14-9e34-6f1230d5076f" (UID: "19f44b1f-4fc6-4f14-9e34-6f1230d5076f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:46:49.901648 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:49.901538    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:49.902989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:49.902967    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-xks-ca" pod "19f44b1f-4fc6-4f14-9e34-6f1230d5076f" (UID: "19f44b1f-4fc6-4f14-9e34-6f1230d5076f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:46:49.903105 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:49.903087    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-kube-api-access-vxxd9" pod "19f44b1f-4fc6-4f14-9e34-6f1230d5076f" (UID: "19f44b1f-4fc6-4f14-9e34-6f1230d5076f"). InnerVolumeSpecName "kube-api-access-vxxd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:50.002632 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.002595    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:50.002632 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.002627    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxxd9\" (UniqueName: \"kubernetes.io/projected/19f44b1f-4fc6-4f14-9e34-6f1230d5076f-kube-api-access-vxxd9\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:50.164580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.164493    4470 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 07:46:50.164580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.164548    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:46:50.334635 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.334600    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"]
Nov 03 07:46:50.335179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.335164    4470 cpu_manager.go:414] "containerMap: removing container" podUID="19f44b1f-4fc6-4f14-9e34-6f1230d5076f" containerName="revoke-keys"
Nov 03 07:46:50.335229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.335181    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f44b1f-4fc6-4f14-9e34-6f1230d5076f" containerName="revoke-keys"
Nov 03 07:46:50.335302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.335290    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f44b1f-4fc6-4f14-9e34-6f1230d5076f" containerName="revoke-keys"
Nov 03 07:46:50.338569 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.338554    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:50.397473 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.397439    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"]
Nov 03 07:46:50.508985 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.508901    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:50.508985 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.508953    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:50.509218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.509066    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q6lr\" (UniqueName: \"kubernetes.io/projected/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-kube-api-access-9q6lr\") pod \"maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:50.592317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.592283    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg" event={"ID":"19f44b1f-4fc6-4f14-9e34-6f1230d5076f","Type":"ContainerDied","Data":"193d609e3ee8cd63b7178dff6f7faa4ce86c608c1778ceec0c76060696729e93"}
Nov 03 07:46:50.592485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.592345    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="193d609e3ee8cd63b7178dff6f7faa4ce86c608c1778ceec0c76060696729e93"
Nov 03 07:46:50.592485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.592298    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"
Nov 03 07:46:50.609663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.609632    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:50.609819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.609692    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:50.609819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.609809    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q6lr\" (UniqueName: \"kubernetes.io/projected/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-kube-api-access-9q6lr\") pod \"maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:50.610292 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.610274    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:50.612258 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.612239    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:50.674487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.674451    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q6lr\" (UniqueName: \"kubernetes.io/projected/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-kube-api-access-9q6lr\") pod \"maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:50.953430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:50.953383    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:51.334342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:51.334317    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"]
Nov 03 07:46:51.335224 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:46:51.335198    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6545b6_052c_43ee_a47d_56f5ecaba4f7.slice/crio-e108c1243818cbb091c2a3b63ad174401445a817ca50c25d3a5400adcca6a98e WatchSource:0}: Error finding container e108c1243818cbb091c2a3b63ad174401445a817ca50c25d3a5400adcca6a98e: Status 404 returned error can't find the container with id e108c1243818cbb091c2a3b63ad174401445a817ca50c25d3a5400adcca6a98e
Nov 03 07:46:51.599064 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:51.598974    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj" event={"ID":"fc6545b6-052c-43ee-a47d-56f5ecaba4f7","Type":"ContainerStarted","Data":"e108c1243818cbb091c2a3b63ad174401445a817ca50c25d3a5400adcca6a98e"}
Nov 03 07:46:52.607025 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:52.606927    4470 generic.go:356] "Generic (PLEG): container finished" podID="fc6545b6-052c-43ee-a47d-56f5ecaba4f7" containerID="d5ea61cb6c7d5198c89fb188ab1f4a4bc7857e4c7961c00ec3ee7b59e140e170" exitCode=0
Nov 03 07:46:52.607491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:52.607025    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj" event={"ID":"fc6545b6-052c-43ee-a47d-56f5ecaba4f7","Type":"ContainerDied","Data":"d5ea61cb6c7d5198c89fb188ab1f4a4bc7857e4c7961c00ec3ee7b59e140e170"}
Nov 03 07:46:53.073210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.073185    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:46:53.139262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139234    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-certs\") pod \"e4219377-453a-4562-8da4-4769cd4ac358\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") "
Nov 03 07:46:53.139489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139283    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-credential-socket\") pod \"e4219377-453a-4562-8da4-4769cd4ac358\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") "
Nov 03 07:46:53.139489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139326    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-istio-token\" (UniqueName: \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-istio-token\") pod \"e4219377-453a-4562-8da4-4769cd4ac358\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") "
Nov 03 07:46:53.139489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139377    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4219377-453a-4562-8da4-4769cd4ac358-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4219377-453a-4562-8da4-4769cd4ac358-istiod-ca-cert\") pod \"e4219377-453a-4562-8da4-4769cd4ac358\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") "
Nov 03 07:46:53.139489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139414    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-data\") pod \"e4219377-453a-4562-8da4-4769cd4ac358\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") "
Nov 03 07:46:53.139489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139444    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-socket\") pod \"e4219377-453a-4562-8da4-4769cd4ac358\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") "
Nov 03 07:46:53.139489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139483    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-kube-api-access-5tmbs\" (UniqueName: \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-kube-api-access-5tmbs\") pod \"e4219377-453a-4562-8da4-4769cd4ac358\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") "
Nov 03 07:46:53.139826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139530    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4219377-453a-4562-8da4-4769cd4ac358-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4219377-453a-4562-8da4-4769cd4ac358-istio-podinfo\") pod \"e4219377-453a-4562-8da4-4769cd4ac358\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") "
Nov 03 07:46:53.139826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139605    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-envoy\") pod \"e4219377-453a-4562-8da4-4769cd4ac358\" (UID: \"e4219377-453a-4562-8da4-4769cd4ac358\") "
Nov 03 07:46:53.139826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139529    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-certs" pod "e4219377-453a-4562-8da4-4769cd4ac358" (UID: "e4219377-453a-4562-8da4-4769cd4ac358"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:53.139826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139711    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-socket" pod "e4219377-453a-4562-8da4-4769cd4ac358" (UID: "e4219377-453a-4562-8da4-4769cd4ac358"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:53.139826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139668    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-credential-socket" pod "e4219377-453a-4562-8da4-4769cd4ac358" (UID: "e4219377-453a-4562-8da4-4769cd4ac358"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:53.140095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.139851    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-data" pod "e4219377-453a-4562-8da4-4769cd4ac358" (UID: "e4219377-453a-4562-8da4-4769cd4ac358"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:53.140434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.140274    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4219377-453a-4562-8da4-4769cd4ac358-istiod-ca-cert" pod "e4219377-453a-4562-8da4-4769cd4ac358" (UID: "e4219377-453a-4562-8da4-4769cd4ac358"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:46:53.140566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.140547    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:53.140642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.140572    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:53.140642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.140587    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:53.140642 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.140601    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:53.142028 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.141999    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-envoy" pod "e4219377-453a-4562-8da4-4769cd4ac358" (UID: "e4219377-453a-4562-8da4-4769cd4ac358"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:46:53.142028 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.142006    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-kube-api-access-5tmbs" pod "e4219377-453a-4562-8da4-4769cd4ac358" (UID: "e4219377-453a-4562-8da4-4769cd4ac358"). InnerVolumeSpecName "kube-api-access-5tmbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:53.142207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.142027    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4219377-453a-4562-8da4-4769cd4ac358-istio-podinfo" pod "e4219377-453a-4562-8da4-4769cd4ac358" (UID: "e4219377-453a-4562-8da4-4769cd4ac358"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:46:53.142207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.142082    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-istio-token" pod "e4219377-453a-4562-8da4-4769cd4ac358" (UID: "e4219377-453a-4562-8da4-4769cd4ac358"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:53.241456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.241382    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:53.241456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.241408    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4219377-453a-4562-8da4-4769cd4ac358-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:53.241456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.241419    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmbs\" (UniqueName: \"kubernetes.io/projected/e4219377-453a-4562-8da4-4769cd4ac358-kube-api-access-5tmbs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:53.241456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.241427    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4219377-453a-4562-8da4-4769cd4ac358-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:53.241456 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.241436    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4219377-453a-4562-8da4-4769cd4ac358-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:53.614994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.614961    4470 generic.go:356] "Generic (PLEG): container finished" podID="e4219377-453a-4562-8da4-4769cd4ac358" containerID="d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c" exitCode=0
Nov 03 07:46:53.615500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.615008    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"
Nov 03 07:46:53.615500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.615049    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" event={"ID":"e4219377-453a-4562-8da4-4769cd4ac358","Type":"ContainerDied","Data":"d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c"}
Nov 03 07:46:53.615500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.615106    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6" event={"ID":"e4219377-453a-4562-8da4-4769cd4ac358","Type":"ContainerDied","Data":"e3b15686571a40eed29b7f0e004315afb05c9230653884f86747718758813033"}
Nov 03 07:46:53.615500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.615147    4470 scope.go:122] "RemoveContainer" containerID="d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c"
Nov 03 07:46:53.626894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.626874    4470 scope.go:122] "RemoveContainer" containerID="bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022"
Nov 03 07:46:53.638787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.638744    4470 scope.go:122] "RemoveContainer" containerID="d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c"
Nov 03 07:46:53.639180 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:46:53.639116    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c\": container with ID starting with d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c not found: ID does not exist" containerID="d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c"
Nov 03 07:46:53.639302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.639190    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c"} err="failed to get container status \"d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c\": rpc error: code = NotFound desc = could not find container \"d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c\": container with ID starting with d11a69bade05aa6129ff1141e018fc4703e0e18d632919bce20d59c4b148f37c not found: ID does not exist"
Nov 03 07:46:53.639302 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.639212    4470 scope.go:122] "RemoveContainer" containerID="bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022"
Nov 03 07:46:53.639568 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:46:53.639549    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022\": container with ID starting with bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022 not found: ID does not exist" containerID="bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022"
Nov 03 07:46:53.639651 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.639578    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022"} err="failed to get container status \"bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022\": rpc error: code = NotFound desc = could not find container \"bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022\": container with ID starting with bb03e5032983a56cf5497051ef320dcea4229c6250c495bf1aeca2250d4fe022 not found: ID does not exist"
Nov 03 07:46:53.766758 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.766732    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:53.787601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.787557    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"]
Nov 03 07:46:53.795014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.794988    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6"]
Nov 03 07:46:53.806351 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.806326    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4219377-453a-4562-8da4-4769cd4ac358" path="/var/lib/kubelet/pods/e4219377-453a-4562-8da4-4769cd4ac358/volumes"
Nov 03 07:46:53.846383 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.846346    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-xks-ca\") pod \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") "
Nov 03 07:46:53.846556 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.846442    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-openshift-ca\") pod \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") "
Nov 03 07:46:53.846556 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.846522    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-kube-api-access-9q6lr\" (UniqueName: \"kubernetes.io/projected/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-kube-api-access-9q6lr\") pod \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\" (UID: \"fc6545b6-052c-43ee-a47d-56f5ecaba4f7\") "
Nov 03 07:46:53.846812 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.846788    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-openshift-ca" pod "fc6545b6-052c-43ee-a47d-56f5ecaba4f7" (UID: "fc6545b6-052c-43ee-a47d-56f5ecaba4f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:46:53.846964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.846945    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:53.848748 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.848722    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-xks-ca" pod "fc6545b6-052c-43ee-a47d-56f5ecaba4f7" (UID: "fc6545b6-052c-43ee-a47d-56f5ecaba4f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:46:53.848748 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.848732    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-kube-api-access-9q6lr" pod "fc6545b6-052c-43ee-a47d-56f5ecaba4f7" (UID: "fc6545b6-052c-43ee-a47d-56f5ecaba4f7"). InnerVolumeSpecName "kube-api-access-9q6lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:46:53.948347 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.948261    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q6lr\" (UniqueName: \"kubernetes.io/projected/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-kube-api-access-9q6lr\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:53.948347 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:53.948293    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc6545b6-052c-43ee-a47d-56f5ecaba4f7-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:46:54.622333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:54.622307    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"
Nov 03 07:46:54.622333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:54.622325    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj" event={"ID":"fc6545b6-052c-43ee-a47d-56f5ecaba4f7","Type":"ContainerDied","Data":"e108c1243818cbb091c2a3b63ad174401445a817ca50c25d3a5400adcca6a98e"}
Nov 03 07:46:54.622892 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:46:54.622385    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e108c1243818cbb091c2a3b63ad174401445a817ca50c25d3a5400adcca6a98e"
Nov 03 07:47:01.898193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:01.898159    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"]
Nov 03 07:47:04.759345 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:47:04.759310    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:47:06.898989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:06.898953    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk" podUID="e916f9c5-3853-42e2-a994-29ab114db2ff" containerName="payload-processing" containerID="cri-o://c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5" gracePeriod=25
Nov 03 07:47:07.183860 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.183826    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:47:07.289629 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.289597    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e916f9c5-3853-42e2-a994-29ab114db2ff-kube-api-access-b492c\" (UniqueName: \"kubernetes.io/projected/e916f9c5-3853-42e2-a994-29ab114db2ff-kube-api-access-b492c\") pod \"e916f9c5-3853-42e2-a994-29ab114db2ff\" (UID: \"e916f9c5-3853-42e2-a994-29ab114db2ff\") "
Nov 03 07:47:07.289845 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.289685    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e916f9c5-3853-42e2-a994-29ab114db2ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e916f9c5-3853-42e2-a994-29ab114db2ff-plugins-config-volume\") pod \"e916f9c5-3853-42e2-a994-29ab114db2ff\" (UID: \"e916f9c5-3853-42e2-a994-29ab114db2ff\") "
Nov 03 07:47:07.290052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.290029    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e916f9c5-3853-42e2-a994-29ab114db2ff-plugins-config-volume" pod "e916f9c5-3853-42e2-a994-29ab114db2ff" (UID: "e916f9c5-3853-42e2-a994-29ab114db2ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:47:07.291989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.291962    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e916f9c5-3853-42e2-a994-29ab114db2ff-kube-api-access-b492c" pod "e916f9c5-3853-42e2-a994-29ab114db2ff" (UID: "e916f9c5-3853-42e2-a994-29ab114db2ff"). InnerVolumeSpecName "kube-api-access-b492c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:47:07.391482 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.391435    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b492c\" (UniqueName: \"kubernetes.io/projected/e916f9c5-3853-42e2-a994-29ab114db2ff-kube-api-access-b492c\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:07.391482 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.391475    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e916f9c5-3853-42e2-a994-29ab114db2ff-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:07.700270 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.700235    4470 generic.go:356] "Generic (PLEG): container finished" podID="e916f9c5-3853-42e2-a994-29ab114db2ff" containerID="c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5" exitCode=0
Nov 03 07:47:07.700483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.700277    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"
Nov 03 07:47:07.700483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.700332    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk" event={"ID":"e916f9c5-3853-42e2-a994-29ab114db2ff","Type":"ContainerDied","Data":"c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5"}
Nov 03 07:47:07.700483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.700427    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk" event={"ID":"e916f9c5-3853-42e2-a994-29ab114db2ff","Type":"ContainerDied","Data":"c4c71b4c4e642ad5f545c9e10c74def122fb65b6e12ac102b30d81067a093aaf"}
Nov 03 07:47:07.700483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.700455    4470 scope.go:122] "RemoveContainer" containerID="c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5"
Nov 03 07:47:07.710468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.710447    4470 scope.go:122] "RemoveContainer" containerID="c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5"
Nov 03 07:47:07.710766 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:47:07.710746    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5\": container with ID starting with c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5 not found: ID does not exist" containerID="c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5"
Nov 03 07:47:07.710826 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.710779    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5"} err="failed to get container status \"c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5\": rpc error: code = NotFound desc = could not find container \"c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5\": container with ID starting with c1758874e60b73b6d0de29cc2ee582cca55db8f07fe14673bd659a74952e74a5 not found: ID does not exist"
Nov 03 07:47:07.883077 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.883049    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"]
Nov 03 07:47:07.931727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:07.931691    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk"]
Nov 03 07:47:08.029936 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:08.029861    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"]
Nov 03 07:47:08.034436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:08.034410    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-669d6b-g98jm"]
Nov 03 07:47:09.805593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:09.805554    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff1e722-5ad1-448c-9339-a26551fe84da" path="/var/lib/kubelet/pods/7ff1e722-5ad1-448c-9339-a26551fe84da/volumes"
Nov 03 07:47:09.805989 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:09.805934    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e916f9c5-3853-42e2-a994-29ab114db2ff" path="/var/lib/kubelet/pods/e916f9c5-3853-42e2-a994-29ab114db2ff/volumes"
Nov 03 07:47:14.731489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:14.731455    4470 scope.go:122] "RemoveContainer" containerID="849c833ea77ac8210ec576b2c1a8e9f20362f9ccfc65c0ad54b8759de0d09193"
Nov 03 07:47:24.617901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:24.617868    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"]
Nov 03 07:47:24.668290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:24.668257    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcdb23-w52cj"]
Nov 03 07:47:25.810502 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:25.810456    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc6545b6-052c-43ee-a47d-56f5ecaba4f7" path="/var/lib/kubelet/pods/fc6545b6-052c-43ee-a47d-56f5ecaba4f7/volumes"
Nov 03 07:47:28.144204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.144173    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"]
Nov 03 07:47:28.144673 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.144393    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerName="istio-proxy" containerID="cri-o://8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51" gracePeriod=30
Nov 03 07:47:28.736705 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.736672    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"]
Nov 03 07:47:28.737226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737200    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy"
Nov 03 07:47:28.737226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737217    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy"
Nov 03 07:47:28.737226 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737227    4470 cpu_manager.go:414] "containerMap: removing container" podUID="fc6545b6-052c-43ee-a47d-56f5ecaba4f7" containerName="revoke-keys"
Nov 03 07:47:28.737345 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737231    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc6545b6-052c-43ee-a47d-56f5ecaba4f7" containerName="revoke-keys"
Nov 03 07:47:28.737345 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737251    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy"
Nov 03 07:47:28.737345 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737256    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy"
Nov 03 07:47:28.737345 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737271    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e916f9c5-3853-42e2-a994-29ab114db2ff" containerName="payload-processing"
Nov 03 07:47:28.737345 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737276    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e916f9c5-3853-42e2-a994-29ab114db2ff" containerName="payload-processing"
Nov 03 07:47:28.737511 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737347    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy"
Nov 03 07:47:28.737511 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737357    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e916f9c5-3853-42e2-a994-29ab114db2ff" containerName="payload-processing"
Nov 03 07:47:28.737511 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737368    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4219377-453a-4562-8da4-4769cd4ac358" containerName="istio-proxy"
Nov 03 07:47:28.737511 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.737374    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc6545b6-052c-43ee-a47d-56f5ecaba4f7" containerName="revoke-keys"
Nov 03 07:47:28.742448 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.742431    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:28.798052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.798018    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:47:28.798052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.798028    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:47:28.798052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.798051    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:47:28.798403 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.798053    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:47:28.799432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.799410    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"]
Nov 03 07:47:28.839645 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.839612    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsj8c\" (UniqueName: \"kubernetes.io/projected/f1474753-9572-4e40-86bf-dc84143e62fa-kube-api-access-zsj8c\") pod \"maas-api-revoke-sub-e1798466-gsfwz\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:28.839829 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.839751    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1798466-gsfwz\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:28.839829 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.839819    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1798466-gsfwz\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:28.941379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.941343    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsj8c\" (UniqueName: \"kubernetes.io/projected/f1474753-9572-4e40-86bf-dc84143e62fa-kube-api-access-zsj8c\") pod \"maas-api-revoke-sub-e1798466-gsfwz\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:28.941571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.941468    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1798466-gsfwz\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:28.941571 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.941540    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1798466-gsfwz\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:28.942309 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.942242    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1798466-gsfwz\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:28.944013 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:28.943991    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1798466-gsfwz\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:29.007126 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:29.007060    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsj8c\" (UniqueName: \"kubernetes.io/projected/f1474753-9572-4e40-86bf-dc84143e62fa-kube-api-access-zsj8c\") pod \"maas-api-revoke-sub-e1798466-gsfwz\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:29.055146 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:29.055106    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:29.432450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:29.432420    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"]
Nov 03 07:47:29.433510 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:47:29.433481    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1474753_9572_4e40_86bf_dc84143e62fa.slice/crio-112dcb5e0bb46ab0bba952303c38eb39bf3225950b23b2ef81c74ff820303874 WatchSource:0}: Error finding container 112dcb5e0bb46ab0bba952303c38eb39bf3225950b23b2ef81c74ff820303874: Status 404 returned error can't find the container with id 112dcb5e0bb46ab0bba952303c38eb39bf3225950b23b2ef81c74ff820303874
Nov 03 07:47:29.839095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:29.839036    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz" event={"ID":"f1474753-9572-4e40-86bf-dc84143e62fa","Type":"ContainerStarted","Data":"582bb28008580f93592c6efb9959a62c94e044bd182241895718c7f18a38a314"}
Nov 03 07:47:29.839095 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:29.839098    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz" event={"ID":"f1474753-9572-4e40-86bf-dc84143e62fa","Type":"ContainerStarted","Data":"112dcb5e0bb46ab0bba952303c38eb39bf3225950b23b2ef81c74ff820303874"}
Nov 03 07:47:29.959828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:29.959777    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz" podStartSLOduration=1.959763604 podStartE2EDuration="1.959763604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:47:29.957589284 +0000 UTC m=+2902.289958282" watchObservedRunningTime="2026-11-03 07:47:29.959763604 +0000 UTC m=+2902.292132597"
Nov 03 07:47:30.854460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:30.854426    4470 generic.go:356] "Generic (PLEG): container finished" podID="f1474753-9572-4e40-86bf-dc84143e62fa" containerID="582bb28008580f93592c6efb9959a62c94e044bd182241895718c7f18a38a314" exitCode=0
Nov 03 07:47:30.856671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:30.854585    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz" event={"ID":"f1474753-9572-4e40-86bf-dc84143e62fa","Type":"ContainerDied","Data":"582bb28008580f93592c6efb9959a62c94e044bd182241895718c7f18a38a314"}
Nov 03 07:47:32.008125 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.008101    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:32.079492 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.079462    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1474753-9572-4e40-86bf-dc84143e62fa-kube-api-access-zsj8c\" (UniqueName: \"kubernetes.io/projected/f1474753-9572-4e40-86bf-dc84143e62fa-kube-api-access-zsj8c\") pod \"f1474753-9572-4e40-86bf-dc84143e62fa\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") "
Nov 03 07:47:32.079677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.079514    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-xks-ca\") pod \"f1474753-9572-4e40-86bf-dc84143e62fa\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") "
Nov 03 07:47:32.079677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.079563    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-openshift-ca\") pod \"f1474753-9572-4e40-86bf-dc84143e62fa\" (UID: \"f1474753-9572-4e40-86bf-dc84143e62fa\") "
Nov 03 07:47:32.079982 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.079957    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-openshift-ca" pod "f1474753-9572-4e40-86bf-dc84143e62fa" (UID: "f1474753-9572-4e40-86bf-dc84143e62fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:47:32.081867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.081836    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1474753-9572-4e40-86bf-dc84143e62fa-kube-api-access-zsj8c" pod "f1474753-9572-4e40-86bf-dc84143e62fa" (UID: "f1474753-9572-4e40-86bf-dc84143e62fa"). InnerVolumeSpecName "kube-api-access-zsj8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:47:32.081994 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.081880    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-xks-ca" pod "f1474753-9572-4e40-86bf-dc84143e62fa" (UID: "f1474753-9572-4e40-86bf-dc84143e62fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:47:32.180929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.180845    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:32.180929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.180871    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1474753-9572-4e40-86bf-dc84143e62fa-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:32.180929 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.180881    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsj8c\" (UniqueName: \"kubernetes.io/projected/f1474753-9572-4e40-86bf-dc84143e62fa-kube-api-access-zsj8c\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:32.867443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.867411    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz" event={"ID":"f1474753-9572-4e40-86bf-dc84143e62fa","Type":"ContainerDied","Data":"112dcb5e0bb46ab0bba952303c38eb39bf3225950b23b2ef81c74ff820303874"}
Nov 03 07:47:32.867664 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.867469    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="112dcb5e0bb46ab0bba952303c38eb39bf3225950b23b2ef81c74ff820303874"
Nov 03 07:47:32.867664 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:32.867421    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"
Nov 03 07:47:33.403448 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.403424    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:47:33.494498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494422    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-credential-socket\") pod \"eb8542d8-8120-4a6c-9457-33812c67e0ae\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") "
Nov 03 07:47:33.494498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494480    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-certs\") pod \"eb8542d8-8120-4a6c-9457-33812c67e0ae\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") "
Nov 03 07:47:33.494724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494575    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-podinfo\") pod \"eb8542d8-8120-4a6c-9457-33812c67e0ae\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") "
Nov 03 07:47:33.494724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494658    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-kube-api-access-szcml\" (UniqueName: \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-kube-api-access-szcml\") pod \"eb8542d8-8120-4a6c-9457-33812c67e0ae\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") "
Nov 03 07:47:33.494724 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494688    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-token\" (UniqueName: \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-token\") pod \"eb8542d8-8120-4a6c-9457-33812c67e0ae\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") "
Nov 03 07:47:33.494899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494720    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-credential-socket" pod "eb8542d8-8120-4a6c-9457-33812c67e0ae" (UID: "eb8542d8-8120-4a6c-9457-33812c67e0ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:47:33.494899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494726    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-socket\") pod \"eb8542d8-8120-4a6c-9457-33812c67e0ae\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") "
Nov 03 07:47:33.494899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494737    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-certs" pod "eb8542d8-8120-4a6c-9457-33812c67e0ae" (UID: "eb8542d8-8120-4a6c-9457-33812c67e0ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:47:33.494899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494769    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-data\") pod \"eb8542d8-8120-4a6c-9457-33812c67e0ae\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") "
Nov 03 07:47:33.494899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494837    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-envoy\") pod \"eb8542d8-8120-4a6c-9457-33812c67e0ae\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") "
Nov 03 07:47:33.495168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494932    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8542d8-8120-4a6c-9457-33812c67e0ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8542d8-8120-4a6c-9457-33812c67e0ae-istiod-ca-cert\") pod \"eb8542d8-8120-4a6c-9457-33812c67e0ae\" (UID: \"eb8542d8-8120-4a6c-9457-33812c67e0ae\") "
Nov 03 07:47:33.495168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.494973    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-socket" pod "eb8542d8-8120-4a6c-9457-33812c67e0ae" (UID: "eb8542d8-8120-4a6c-9457-33812c67e0ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:47:33.495297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.495198    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-data" pod "eb8542d8-8120-4a6c-9457-33812c67e0ae" (UID: "eb8542d8-8120-4a6c-9457-33812c67e0ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:47:33.495364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.495294    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8542d8-8120-4a6c-9457-33812c67e0ae-istiod-ca-cert" pod "eb8542d8-8120-4a6c-9457-33812c67e0ae" (UID: "eb8542d8-8120-4a6c-9457-33812c67e0ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:47:33.495611 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.495587    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:33.495678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.495612    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:33.495678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.495630    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:33.495678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.495644    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8542d8-8120-4a6c-9457-33812c67e0ae-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:33.495678 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.495658    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:33.497091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.497063    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-podinfo" pod "eb8542d8-8120-4a6c-9457-33812c67e0ae" (UID: "eb8542d8-8120-4a6c-9457-33812c67e0ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:47:33.497091 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.497093    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-kube-api-access-szcml" pod "eb8542d8-8120-4a6c-9457-33812c67e0ae" (UID: "eb8542d8-8120-4a6c-9457-33812c67e0ae"). InnerVolumeSpecName "kube-api-access-szcml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:47:33.497340 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.497178    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-token" pod "eb8542d8-8120-4a6c-9457-33812c67e0ae" (UID: "eb8542d8-8120-4a6c-9457-33812c67e0ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:47:33.497510 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.497490    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-envoy" pod "eb8542d8-8120-4a6c-9457-33812c67e0ae" (UID: "eb8542d8-8120-4a6c-9457-33812c67e0ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:47:33.597575 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.597542    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:33.597575 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.597578    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:33.597777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.597593    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szcml\" (UniqueName: \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-kube-api-access-szcml\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:33.597777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.597607    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8542d8-8120-4a6c-9457-33812c67e0ae-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:33.874861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.874827    4470 generic.go:356] "Generic (PLEG): container finished" podID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerID="8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51" exitCode=0
Nov 03 07:47:33.875058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.874900    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" event={"ID":"eb8542d8-8120-4a6c-9457-33812c67e0ae","Type":"ContainerDied","Data":"8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51"}
Nov 03 07:47:33.875058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.874961    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7" event={"ID":"eb8542d8-8120-4a6c-9457-33812c67e0ae","Type":"ContainerDied","Data":"28a84ae2975448b4c060bc39db415774c2073b817c2dfc94cc50c1ac7ad5d237"}
Nov 03 07:47:33.875058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.874901    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"
Nov 03 07:47:33.875058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.874980    4470 scope.go:122] "RemoveContainer" containerID="8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51"
Nov 03 07:47:33.887902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.887876    4470 scope.go:122] "RemoveContainer" containerID="8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51"
Nov 03 07:47:33.888190 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:47:33.888170    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51\": container with ID starting with 8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51 not found: ID does not exist" containerID="8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51"
Nov 03 07:47:33.888253 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.888199    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51"} err="failed to get container status \"8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51\": rpc error: code = NotFound desc = could not find container \"8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51\": container with ID starting with 8304c26043fb1fbc9b226d25c551a96ca085e3fb14208614fb396f071d83bf51 not found: ID does not exist"
Nov 03 07:47:33.910356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.910323    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"]
Nov 03 07:47:33.910798 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.910786    4470 cpu_manager.go:414] "containerMap: removing container" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerName="istio-proxy"
Nov 03 07:47:33.910843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.910799    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerName="istio-proxy"
Nov 03 07:47:33.910843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.910813    4470 cpu_manager.go:414] "containerMap: removing container" podUID="f1474753-9572-4e40-86bf-dc84143e62fa" containerName="revoke-keys"
Nov 03 07:47:33.910843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.910818    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1474753-9572-4e40-86bf-dc84143e62fa" containerName="revoke-keys"
Nov 03 07:47:33.910961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.910883    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" containerName="istio-proxy"
Nov 03 07:47:33.910961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.910892    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1474753-9572-4e40-86bf-dc84143e62fa" containerName="revoke-keys"
Nov 03 07:47:33.915358 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.915332    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:33.971404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.971364    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:47:33.971590 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.971448    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:47:33.971590 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.971456    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:47:33.973973 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.973949    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"]
Nov 03 07:47:33.980332 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:33.980313    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:47:34.002038 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.002011    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:34.002216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.002063    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9tw\" (UniqueName: \"kubernetes.io/projected/7750ebd0-34bd-4b16-a462-3e43d6464421-kube-api-access-nj9tw\") pod \"maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:34.002273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.002224    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:34.103300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.103217    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:34.103300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.103262    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9tw\" (UniqueName: \"kubernetes.io/projected/7750ebd0-34bd-4b16-a462-3e43d6464421-kube-api-access-nj9tw\") pod \"maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:34.103513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.103315    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:34.103984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.103965    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:34.105728 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.105709    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:34.170124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.170091    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"]
Nov 03 07:47:34.172276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.172249    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9tw\" (UniqueName: \"kubernetes.io/projected/7750ebd0-34bd-4b16-a462-3e43d6464421-kube-api-access-nj9tw\") pod \"maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:34.173959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.173936    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7"]
Nov 03 07:47:34.232184 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.232153    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:34.608867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.608840    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"]
Nov 03 07:47:34.610572 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:47:34.610545    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7750ebd0_34bd_4b16_a462_3e43d6464421.slice/crio-aefbe126423daf1dfc72de4175c2630d878542a8c44196b164bc3863e237f79a WatchSource:0}: Error finding container aefbe126423daf1dfc72de4175c2630d878542a8c44196b164bc3863e237f79a: Status 404 returned error can't find the container with id aefbe126423daf1dfc72de4175c2630d878542a8c44196b164bc3863e237f79a
Nov 03 07:47:34.760207 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:47:34.760176    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:47:34.883478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.883446    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv" event={"ID":"7750ebd0-34bd-4b16-a462-3e43d6464421","Type":"ContainerStarted","Data":"11046d221bfec023fed731584bd88a41072ad66ecfa8d9bf98e7c0426e0fe24a"}
Nov 03 07:47:34.883649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:34.883506    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv" event={"ID":"7750ebd0-34bd-4b16-a462-3e43d6464421","Type":"ContainerStarted","Data":"aefbe126423daf1dfc72de4175c2630d878542a8c44196b164bc3863e237f79a"}
Nov 03 07:47:35.004143 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:35.004088    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv" podStartSLOduration=2.004072725 podStartE2EDuration="2.004072725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:47:35.002966279 +0000 UTC m=+2907.335335274" watchObservedRunningTime="2026-11-03 07:47:35.004072725 +0000 UTC m=+2907.336441755"
Nov 03 07:47:35.806494 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:35.806398    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8542d8-8120-4a6c-9457-33812c67e0ae" path="/var/lib/kubelet/pods/eb8542d8-8120-4a6c-9457-33812c67e0ae/volumes"
Nov 03 07:47:35.890447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:35.890401    4470 generic.go:356] "Generic (PLEG): container finished" podID="7750ebd0-34bd-4b16-a462-3e43d6464421" containerID="11046d221bfec023fed731584bd88a41072ad66ecfa8d9bf98e7c0426e0fe24a" exitCode=0
Nov 03 07:47:35.890627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:35.890485    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv" event={"ID":"7750ebd0-34bd-4b16-a462-3e43d6464421","Type":"ContainerDied","Data":"11046d221bfec023fed731584bd88a41072ad66ecfa8d9bf98e7c0426e0fe24a"}
Nov 03 07:47:37.039014 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.038988    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:37.136308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.136276    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-openshift-ca\") pod \"7750ebd0-34bd-4b16-a462-3e43d6464421\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") "
Nov 03 07:47:37.136470 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.136343    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7750ebd0-34bd-4b16-a462-3e43d6464421-kube-api-access-nj9tw\" (UniqueName: \"kubernetes.io/projected/7750ebd0-34bd-4b16-a462-3e43d6464421-kube-api-access-nj9tw\") pod \"7750ebd0-34bd-4b16-a462-3e43d6464421\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") "
Nov 03 07:47:37.136470 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.136409    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-xks-ca\") pod \"7750ebd0-34bd-4b16-a462-3e43d6464421\" (UID: \"7750ebd0-34bd-4b16-a462-3e43d6464421\") "
Nov 03 07:47:37.136689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.136631    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-openshift-ca" pod "7750ebd0-34bd-4b16-a462-3e43d6464421" (UID: "7750ebd0-34bd-4b16-a462-3e43d6464421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:47:37.136842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.136823    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:37.138670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.138643    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7750ebd0-34bd-4b16-a462-3e43d6464421-kube-api-access-nj9tw" pod "7750ebd0-34bd-4b16-a462-3e43d6464421" (UID: "7750ebd0-34bd-4b16-a462-3e43d6464421"). InnerVolumeSpecName "kube-api-access-nj9tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:47:37.138787 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.138762    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-xks-ca" pod "7750ebd0-34bd-4b16-a462-3e43d6464421" (UID: "7750ebd0-34bd-4b16-a462-3e43d6464421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:47:37.237824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.237743    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj9tw\" (UniqueName: \"kubernetes.io/projected/7750ebd0-34bd-4b16-a462-3e43d6464421-kube-api-access-nj9tw\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:37.237824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.237772    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7750ebd0-34bd-4b16-a462-3e43d6464421-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:37.903812 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.903783    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"
Nov 03 07:47:37.903812 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.903802    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv" event={"ID":"7750ebd0-34bd-4b16-a462-3e43d6464421","Type":"ContainerDied","Data":"aefbe126423daf1dfc72de4175c2630d878542a8c44196b164bc3863e237f79a"}
Nov 03 07:47:37.904079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:37.903843    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aefbe126423daf1dfc72de4175c2630d878542a8c44196b164bc3863e237f79a"
Nov 03 07:47:38.326522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:38.326484    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"]
Nov 03 07:47:45.837192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.837160    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"]
Nov 03 07:47:45.837638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.837623    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7750ebd0-34bd-4b16-a462-3e43d6464421" containerName="revoke-keys"
Nov 03 07:47:45.837680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.837640    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7750ebd0-34bd-4b16-a462-3e43d6464421" containerName="revoke-keys"
Nov 03 07:47:45.837755 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.837746    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7750ebd0-34bd-4b16-a462-3e43d6464421" containerName="revoke-keys"
Nov 03 07:47:45.841049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.841035    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:45.900711 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.900680    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"]
Nov 03 07:47:45.915416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.915380    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:47:45.915600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.915421    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:47:45.915600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.915508    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:47:45.915600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.915542    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:47:45.935198 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.935168    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:45.935346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.935225    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4rjg\" (UniqueName: \"kubernetes.io/projected/cc159fa6-7ac1-4c15-8706-79b912e56e8e-kube-api-access-j4rjg\") pod \"maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:45.935346 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:45.935258    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:46.036315 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:46.036281    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:46.036486 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:46.036333    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rjg\" (UniqueName: \"kubernetes.io/projected/cc159fa6-7ac1-4c15-8706-79b912e56e8e-kube-api-access-j4rjg\") pod \"maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:46.036486 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:46.036355    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:46.037067 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:46.037004    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:46.038801 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:46.038778    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:46.100564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:46.100500    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4rjg\" (UniqueName: \"kubernetes.io/projected/cc159fa6-7ac1-4c15-8706-79b912e56e8e-kube-api-access-j4rjg\") pod \"maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:46.153979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:46.153950    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:46.530794 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:46.530762    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"]
Nov 03 07:47:46.532179 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:47:46.532150    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc159fa6_7ac1_4c15_8706_79b912e56e8e.slice/crio-5432e91a08fae4b63276df136f2c34369494d80f94bc1190697bece4d0503f67 WatchSource:0}: Error finding container 5432e91a08fae4b63276df136f2c34369494d80f94bc1190697bece4d0503f67: Status 404 returned error can't find the container with id 5432e91a08fae4b63276df136f2c34369494d80f94bc1190697bece4d0503f67
Nov 03 07:47:46.971318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:46.971279    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq" event={"ID":"cc159fa6-7ac1-4c15-8706-79b912e56e8e","Type":"ContainerStarted","Data":"99ed64ea7f9f9434bcaf4844946958825372110d79d20183646485ba1cc5972f"}
Nov 03 07:47:46.971737 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:46.971339    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq" event={"ID":"cc159fa6-7ac1-4c15-8706-79b912e56e8e","Type":"ContainerStarted","Data":"5432e91a08fae4b63276df136f2c34369494d80f94bc1190697bece4d0503f67"}
Nov 03 07:47:47.124031 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:47.123925    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq" podStartSLOduration=2.123908012 podStartE2EDuration="2.123908012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:47:47.123511355 +0000 UTC m=+2919.455880352" watchObservedRunningTime="2026-11-03 07:47:47.123908012 +0000 UTC m=+2919.456277024"
Nov 03 07:47:47.979315 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:47.979271    4470 generic.go:356] "Generic (PLEG): container finished" podID="cc159fa6-7ac1-4c15-8706-79b912e56e8e" containerID="99ed64ea7f9f9434bcaf4844946958825372110d79d20183646485ba1cc5972f" exitCode=0
Nov 03 07:47:47.979772 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:47.979357    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq" event={"ID":"cc159fa6-7ac1-4c15-8706-79b912e56e8e","Type":"ContainerDied","Data":"99ed64ea7f9f9434bcaf4844946958825372110d79d20183646485ba1cc5972f"}
Nov 03 07:47:49.120296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.120266    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"]
Nov 03 07:47:49.151566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.151541    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:49.172402 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.172356    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-xks-ca\") pod \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") "
Nov 03 07:47:49.172568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.172437    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-openshift-ca\") pod \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") "
Nov 03 07:47:49.172626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.172589    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc159fa6-7ac1-4c15-8706-79b912e56e8e-kube-api-access-j4rjg\" (UniqueName: \"kubernetes.io/projected/cc159fa6-7ac1-4c15-8706-79b912e56e8e-kube-api-access-j4rjg\") pod \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\" (UID: \"cc159fa6-7ac1-4c15-8706-79b912e56e8e\") "
Nov 03 07:47:49.175992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.175957    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc159fa6-7ac1-4c15-8706-79b912e56e8e-kube-api-access-j4rjg" pod "cc159fa6-7ac1-4c15-8706-79b912e56e8e" (UID: "cc159fa6-7ac1-4c15-8706-79b912e56e8e"). InnerVolumeSpecName "kube-api-access-j4rjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:47:49.176855 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.176830    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-openshift-ca" pod "cc159fa6-7ac1-4c15-8706-79b912e56e8e" (UID: "cc159fa6-7ac1-4c15-8706-79b912e56e8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:47:49.178453 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.178425    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-xks-ca" pod "cc159fa6-7ac1-4c15-8706-79b912e56e8e" (UID: "cc159fa6-7ac1-4c15-8706-79b912e56e8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:47:49.274436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.274356    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4rjg\" (UniqueName: \"kubernetes.io/projected/cc159fa6-7ac1-4c15-8706-79b912e56e8e-kube-api-access-j4rjg\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:49.274436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.274388    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:49.274436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.274400    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc159fa6-7ac1-4c15-8706-79b912e56e8e-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:49.993286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.993260    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"
Nov 03 07:47:49.993439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.993323    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq" event={"ID":"cc159fa6-7ac1-4c15-8706-79b912e56e8e","Type":"ContainerDied","Data":"5432e91a08fae4b63276df136f2c34369494d80f94bc1190697bece4d0503f67"}
Nov 03 07:47:49.993439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:49.993375    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5432e91a08fae4b63276df136f2c34369494d80f94bc1190697bece4d0503f67"
Nov 03 07:47:53.353734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.353697    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main" containerID="cri-o://651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa" gracePeriod=45
Nov 03 07:47:53.608433 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.608365    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:47:53.718353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.718316    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-home\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-home\") pod \"4d2a3693-ff83-4324-ba34-5722fdb42861\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") "
Nov 03 07:47:53.718539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.718409    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-tmp-dir\") pod \"4d2a3693-ff83-4324-ba34-5722fdb42861\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") "
Nov 03 07:47:53.718539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.718469    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-dshm\") pod \"4d2a3693-ff83-4324-ba34-5722fdb42861\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") "
Nov 03 07:47:53.718539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.718508    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d2a3693-ff83-4324-ba34-5722fdb42861-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2a3693-ff83-4324-ba34-5722fdb42861-tls-certs\") pod \"4d2a3693-ff83-4324-ba34-5722fdb42861\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") "
Nov 03 07:47:53.718658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.718618    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2a3693-ff83-4324-ba34-5722fdb42861-kube-api-access-5zmvg\" (UniqueName: \"kubernetes.io/projected/4d2a3693-ff83-4324-ba34-5722fdb42861-kube-api-access-5zmvg\") pod \"4d2a3693-ff83-4324-ba34-5722fdb42861\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") "
Nov 03 07:47:53.718658 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.718635    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-home" pod "4d2a3693-ff83-4324-ba34-5722fdb42861" (UID: "4d2a3693-ff83-4324-ba34-5722fdb42861"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:47:53.718775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.718661    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-model-cache\") pod \"4d2a3693-ff83-4324-ba34-5722fdb42861\" (UID: \"4d2a3693-ff83-4324-ba34-5722fdb42861\") "
Nov 03 07:47:53.718775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.718675    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-tmp-dir" pod "4d2a3693-ff83-4324-ba34-5722fdb42861" (UID: "4d2a3693-ff83-4324-ba34-5722fdb42861"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:47:53.718907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.718874    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-model-cache" pod "4d2a3693-ff83-4324-ba34-5722fdb42861" (UID: "4d2a3693-ff83-4324-ba34-5722fdb42861"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:47:53.719003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.718986    4470 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-model-cache\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:53.719052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.719007    4470 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-home\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:53.719052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.719018    4470 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-tmp-dir\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:53.720924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.720903    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-dshm" pod "4d2a3693-ff83-4324-ba34-5722fdb42861" (UID: "4d2a3693-ff83-4324-ba34-5722fdb42861"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:47:53.721017 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.720987    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2a3693-ff83-4324-ba34-5722fdb42861-kube-api-access-5zmvg" pod "4d2a3693-ff83-4324-ba34-5722fdb42861" (UID: "4d2a3693-ff83-4324-ba34-5722fdb42861"). InnerVolumeSpecName "kube-api-access-5zmvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:47:53.721058 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.721008    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d2a3693-ff83-4324-ba34-5722fdb42861-tls-certs" pod "4d2a3693-ff83-4324-ba34-5722fdb42861" (UID: "4d2a3693-ff83-4324-ba34-5722fdb42861"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:47:53.819622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.819595    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zmvg\" (UniqueName: \"kubernetes.io/projected/4d2a3693-ff83-4324-ba34-5722fdb42861-kube-api-access-5zmvg\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:53.819757 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.819623    4470 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2a3693-ff83-4324-ba34-5722fdb42861-dshm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:53.819757 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:53.819644    4470 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2a3693-ff83-4324-ba34-5722fdb42861-tls-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:54.018019 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.017936    4470 generic.go:356] "Generic (PLEG): container finished" podID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerID="651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa" exitCode=0
Nov 03 07:47:54.018019 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.017979    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"
Nov 03 07:47:54.018265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.018018    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" event={"ID":"4d2a3693-ff83-4324-ba34-5722fdb42861","Type":"ContainerDied","Data":"651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa"}
Nov 03 07:47:54.018265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.018062    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct" event={"ID":"4d2a3693-ff83-4324-ba34-5722fdb42861","Type":"ContainerDied","Data":"a730c6f9ccd5a1406e448b2e48c91d536143a2cf7681baff0dca43f26eb1a397"}
Nov 03 07:47:54.018265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.018081    4470 scope.go:122] "RemoveContainer" containerID="651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa"
Nov 03 07:47:54.027601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.027572    4470 scope.go:122] "RemoveContainer" containerID="651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa"
Nov 03 07:47:54.027874 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:47:54.027854    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa\": container with ID starting with 651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa not found: ID does not exist" containerID="651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa"
Nov 03 07:47:54.027915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.027883    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa"} err="failed to get container status \"651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa\": rpc error: code = NotFound desc = could not find container \"651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa\": container with ID starting with 651bcac1643246513f0dd7e140f6878be30262187e909982182e5c4ffc4c00aa not found: ID does not exist"
Nov 03 07:47:54.120669 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.120630    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4" podUID="91d2c70b-2ef0-45c2-9bce-8a02d58db1c9" containerName="payload-processing" containerID="cri-o://e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4" gracePeriod=25
Nov 03 07:47:54.194009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.193970    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"]
Nov 03 07:47:54.198531 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.198501    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-2-w0-6606d6-kserve-d7f75f966-pb2ct"]
Nov 03 07:47:54.362823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.362799    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:47:54.426185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.426126    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-kube-api-access-vvzr2\" (UniqueName: \"kubernetes.io/projected/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-kube-api-access-vvzr2\") pod \"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9\" (UID: \"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9\") "
Nov 03 07:47:54.426365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.426252    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-plugins-config-volume\") pod \"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9\" (UID: \"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9\") "
Nov 03 07:47:54.426600 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.426577    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-plugins-config-volume" pod "91d2c70b-2ef0-45c2-9bce-8a02d58db1c9" (UID: "91d2c70b-2ef0-45c2-9bce-8a02d58db1c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:47:54.428567 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.428547    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-kube-api-access-vvzr2" pod "91d2c70b-2ef0-45c2-9bce-8a02d58db1c9" (UID: "91d2c70b-2ef0-45c2-9bce-8a02d58db1c9"). InnerVolumeSpecName "kube-api-access-vvzr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:47:54.528020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.527986    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvzr2\" (UniqueName: \"kubernetes.io/projected/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-kube-api-access-vvzr2\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:54.528020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:54.528016    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:47:55.025539 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:55.025503    4470 generic.go:356] "Generic (PLEG): container finished" podID="91d2c70b-2ef0-45c2-9bce-8a02d58db1c9" containerID="e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4" exitCode=0
Nov 03 07:47:55.025740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:55.025547    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"
Nov 03 07:47:55.025740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:55.025593    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4" event={"ID":"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9","Type":"ContainerDied","Data":"e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4"}
Nov 03 07:47:55.025740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:55.025659    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4" event={"ID":"91d2c70b-2ef0-45c2-9bce-8a02d58db1c9","Type":"ContainerDied","Data":"86fe6ece3eee9c7ffc77cbc706df6b445be7b789df5bebe225dc7d28ab7d4ba0"}
Nov 03 07:47:55.025740 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:55.025691    4470 scope.go:122] "RemoveContainer" containerID="e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4"
Nov 03 07:47:55.035901 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:55.035883    4470 scope.go:122] "RemoveContainer" containerID="e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4"
Nov 03 07:47:55.036253 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:47:55.036234    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4\": container with ID starting with e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4 not found: ID does not exist" containerID="e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4"
Nov 03 07:47:55.036306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:55.036262    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4"} err="failed to get container status \"e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4\": rpc error: code = NotFound desc = could not find container \"e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4\": container with ID starting with e7cbcc55d398171bc52fd4c5f11172bbe538910071ccd3e116827bd668be31b4 not found: ID does not exist"
Nov 03 07:47:55.197201 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:55.197166    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"]
Nov 03 07:47:55.204738 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:55.204712    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4"]
Nov 03 07:47:55.805483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:55.805454    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" path="/var/lib/kubelet/pods/4d2a3693-ff83-4324-ba34-5722fdb42861/volumes"
Nov 03 07:47:55.805962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:47:55.805831    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d2c70b-2ef0-45c2-9bce-8a02d58db1c9" path="/var/lib/kubelet/pods/91d2c70b-2ef0-45c2-9bce-8a02d58db1c9/volumes"
Nov 03 07:48:00.364936 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.364901    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"]
Nov 03 07:48:00.365352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.365342    4470 cpu_manager.go:414] "containerMap: removing container" podUID="91d2c70b-2ef0-45c2-9bce-8a02d58db1c9" containerName="payload-processing"
Nov 03 07:48:00.365401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.365354    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d2c70b-2ef0-45c2-9bce-8a02d58db1c9" containerName="payload-processing"
Nov 03 07:48:00.365401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.365378    4470 cpu_manager.go:414] "containerMap: removing container" podUID="cc159fa6-7ac1-4c15-8706-79b912e56e8e" containerName="revoke-keys"
Nov 03 07:48:00.365401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.365383    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc159fa6-7ac1-4c15-8706-79b912e56e8e" containerName="revoke-keys"
Nov 03 07:48:00.365401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.365391    4470 cpu_manager.go:414] "containerMap: removing container" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main"
Nov 03 07:48:00.365401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.365396    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main"
Nov 03 07:48:00.365562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.365459    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc159fa6-7ac1-4c15-8706-79b912e56e8e" containerName="revoke-keys"
Nov 03 07:48:00.365562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.365466    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2a3693-ff83-4324-ba34-5722fdb42861" containerName="main"
Nov 03 07:48:00.365562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.365476    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d2c70b-2ef0-45c2-9bce-8a02d58db1c9" containerName="payload-processing"
Nov 03 07:48:00.369980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.369963    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:00.421773 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.421733    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"]
Nov 03 07:48:00.425789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.425741    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:00.469911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.469879    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:48:00.470118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.469978    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:48:00.473190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.473170    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"]
Nov 03 07:48:00.475721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.475700    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:48:00.475849 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.475800    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:48:00.491533 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.491507    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-647c1c93-6lgw7\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:00.491684 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.491547    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eded570c-l7b7v\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:00.491684 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.491579    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-647c1c93-6lgw7\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:00.491809 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.491786    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmpld\" (UniqueName: \"kubernetes.io/projected/602e922a-3f3e-4388-a0f5-00d70b163d24-kube-api-access-gmpld\") pod \"maas-api-revoke-sub-eded570c-l7b7v\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:00.491866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.491825    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnkg\" (UniqueName: \"kubernetes.io/projected/c670d8f1-5cc5-4519-b051-c7faeaeabb37-kube-api-access-tbnkg\") pod \"maas-api-revoke-sub-647c1c93-6lgw7\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:00.491866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.491850    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eded570c-l7b7v\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:00.536159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.531635    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"]
Nov 03 07:48:00.592804 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.592769    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnkg\" (UniqueName: \"kubernetes.io/projected/c670d8f1-5cc5-4519-b051-c7faeaeabb37-kube-api-access-tbnkg\") pod \"maas-api-revoke-sub-647c1c93-6lgw7\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:00.592987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.592811    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eded570c-l7b7v\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:00.592987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.592858    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-647c1c93-6lgw7\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:00.592987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.592880    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eded570c-l7b7v\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:00.592987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.592904    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-647c1c93-6lgw7\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:00.593229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.593016    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmpld\" (UniqueName: \"kubernetes.io/projected/602e922a-3f3e-4388-a0f5-00d70b163d24-kube-api-access-gmpld\") pod \"maas-api-revoke-sub-eded570c-l7b7v\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:00.593756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.593678    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eded570c-l7b7v\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:00.593756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.593683    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-647c1c93-6lgw7\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:00.595472 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.595453    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eded570c-l7b7v\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:00.595561 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.595452    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-647c1c93-6lgw7\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:00.657847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.657776    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnkg\" (UniqueName: \"kubernetes.io/projected/c670d8f1-5cc5-4519-b051-c7faeaeabb37-kube-api-access-tbnkg\") pod \"maas-api-revoke-sub-647c1c93-6lgw7\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:00.660428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.660405    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmpld\" (UniqueName: \"kubernetes.io/projected/602e922a-3f3e-4388-a0f5-00d70b163d24-kube-api-access-gmpld\") pod \"maas-api-revoke-sub-eded570c-l7b7v\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:00.683273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.683242    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:00.739907 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:00.739872    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:01.073212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:01.073186    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"]
Nov 03 07:48:01.074528 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:48:01.074505    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602e922a_3f3e_4388_a0f5_00d70b163d24.slice/crio-70a14375e83e21a7116b90e8cdc1ab72359e21c3acb77bd84615728cfa106686 WatchSource:0}: Error finding container 70a14375e83e21a7116b90e8cdc1ab72359e21c3acb77bd84615728cfa106686: Status 404 returned error can't find the container with id 70a14375e83e21a7116b90e8cdc1ab72359e21c3acb77bd84615728cfa106686
Nov 03 07:48:01.123515 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:01.123491    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"]
Nov 03 07:48:01.124484 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:48:01.124460    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc670d8f1_5cc5_4519_b051_c7faeaeabb37.slice/crio-3eb17b4c98ed796cc48bedbae08d8f1d686cf0146baca2ec7e4bc4da16a7b90a WatchSource:0}: Error finding container 3eb17b4c98ed796cc48bedbae08d8f1d686cf0146baca2ec7e4bc4da16a7b90a: Status 404 returned error can't find the container with id 3eb17b4c98ed796cc48bedbae08d8f1d686cf0146baca2ec7e4bc4da16a7b90a
Nov 03 07:48:02.074681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:02.074644    4470 generic.go:356] "Generic (PLEG): container finished" podID="602e922a-3f3e-4388-a0f5-00d70b163d24" containerID="005b770539e13a28117be9615eeb9537a18dd5a0e67883128878fcb774b92b5c" exitCode=0
Nov 03 07:48:02.075151 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:02.074726    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v" event={"ID":"602e922a-3f3e-4388-a0f5-00d70b163d24","Type":"ContainerDied","Data":"005b770539e13a28117be9615eeb9537a18dd5a0e67883128878fcb774b92b5c"}
Nov 03 07:48:02.075151 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:02.074773    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v" event={"ID":"602e922a-3f3e-4388-a0f5-00d70b163d24","Type":"ContainerStarted","Data":"70a14375e83e21a7116b90e8cdc1ab72359e21c3acb77bd84615728cfa106686"}
Nov 03 07:48:02.076495 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:02.076474    4470 generic.go:356] "Generic (PLEG): container finished" podID="c670d8f1-5cc5-4519-b051-c7faeaeabb37" containerID="9f00cba410b4d3877f3e9b502e1e4e6d27dbf81462fb5469caadc253c363dd8e" exitCode=0
Nov 03 07:48:02.076614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:02.076524    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7" event={"ID":"c670d8f1-5cc5-4519-b051-c7faeaeabb37","Type":"ContainerDied","Data":"9f00cba410b4d3877f3e9b502e1e4e6d27dbf81462fb5469caadc253c363dd8e"}
Nov 03 07:48:02.076614 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:02.076549    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7" event={"ID":"c670d8f1-5cc5-4519-b051-c7faeaeabb37","Type":"ContainerStarted","Data":"3eb17b4c98ed796cc48bedbae08d8f1d686cf0146baca2ec7e4bc4da16a7b90a"}
Nov 03 07:48:02.707799 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:02.707761    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"]
Nov 03 07:48:02.711887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:02.711858    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50ae96-dqbcq"]
Nov 03 07:48:03.252205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.252177    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:03.258419 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.258395    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:03.327442 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.327403    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c670d8f1-5cc5-4519-b051-c7faeaeabb37-kube-api-access-tbnkg\" (UniqueName: \"kubernetes.io/projected/c670d8f1-5cc5-4519-b051-c7faeaeabb37-kube-api-access-tbnkg\") pod \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") "
Nov 03 07:48:03.327656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.327452    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-openshift-ca\") pod \"602e922a-3f3e-4388-a0f5-00d70b163d24\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") "
Nov 03 07:48:03.327656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.327497    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602e922a-3f3e-4388-a0f5-00d70b163d24-kube-api-access-gmpld\" (UniqueName: \"kubernetes.io/projected/602e922a-3f3e-4388-a0f5-00d70b163d24-kube-api-access-gmpld\") pod \"602e922a-3f3e-4388-a0f5-00d70b163d24\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") "
Nov 03 07:48:03.327656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.327546    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-xks-ca\") pod \"602e922a-3f3e-4388-a0f5-00d70b163d24\" (UID: \"602e922a-3f3e-4388-a0f5-00d70b163d24\") "
Nov 03 07:48:03.327656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.327617    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-xks-ca\") pod \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") "
Nov 03 07:48:03.327876 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.327707    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-openshift-ca\") pod \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\" (UID: \"c670d8f1-5cc5-4519-b051-c7faeaeabb37\") "
Nov 03 07:48:03.327930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.327871    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-openshift-ca" pod "602e922a-3f3e-4388-a0f5-00d70b163d24" (UID: "602e922a-3f3e-4388-a0f5-00d70b163d24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:48:03.328187 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.328113    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-openshift-ca" pod "c670d8f1-5cc5-4519-b051-c7faeaeabb37" (UID: "c670d8f1-5cc5-4519-b051-c7faeaeabb37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:48:03.328489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.328469    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:03.328553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.328496    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:03.329955 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.329925    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-xks-ca" pod "c670d8f1-5cc5-4519-b051-c7faeaeabb37" (UID: "c670d8f1-5cc5-4519-b051-c7faeaeabb37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:48:03.330064 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.329976    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602e922a-3f3e-4388-a0f5-00d70b163d24-kube-api-access-gmpld" pod "602e922a-3f3e-4388-a0f5-00d70b163d24" (UID: "602e922a-3f3e-4388-a0f5-00d70b163d24"). InnerVolumeSpecName "kube-api-access-gmpld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:48:03.330353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.330324    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-xks-ca" pod "602e922a-3f3e-4388-a0f5-00d70b163d24" (UID: "602e922a-3f3e-4388-a0f5-00d70b163d24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:48:03.330353 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.330340    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c670d8f1-5cc5-4519-b051-c7faeaeabb37-kube-api-access-tbnkg" pod "c670d8f1-5cc5-4519-b051-c7faeaeabb37" (UID: "c670d8f1-5cc5-4519-b051-c7faeaeabb37"). InnerVolumeSpecName "kube-api-access-tbnkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:48:03.429943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.429834    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbnkg\" (UniqueName: \"kubernetes.io/projected/c670d8f1-5cc5-4519-b051-c7faeaeabb37-kube-api-access-tbnkg\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:03.429943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.429869    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmpld\" (UniqueName: \"kubernetes.io/projected/602e922a-3f3e-4388-a0f5-00d70b163d24-kube-api-access-gmpld\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:03.429943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.429881    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602e922a-3f3e-4388-a0f5-00d70b163d24-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:03.429943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.429891    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c670d8f1-5cc5-4519-b051-c7faeaeabb37-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:03.806286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:03.806211    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc159fa6-7ac1-4c15-8706-79b912e56e8e" path="/var/lib/kubelet/pods/cc159fa6-7ac1-4c15-8706-79b912e56e8e/volumes"
Nov 03 07:48:04.097485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:04.097453    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"
Nov 03 07:48:04.097485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:04.097477    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7" event={"ID":"c670d8f1-5cc5-4519-b051-c7faeaeabb37","Type":"ContainerDied","Data":"3eb17b4c98ed796cc48bedbae08d8f1d686cf0146baca2ec7e4bc4da16a7b90a"}
Nov 03 07:48:04.097712 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:04.097529    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eb17b4c98ed796cc48bedbae08d8f1d686cf0146baca2ec7e4bc4da16a7b90a"
Nov 03 07:48:04.098790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:04.098774    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"
Nov 03 07:48:04.098909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:04.098816    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v" event={"ID":"602e922a-3f3e-4388-a0f5-00d70b163d24","Type":"ContainerDied","Data":"70a14375e83e21a7116b90e8cdc1ab72359e21c3acb77bd84615728cfa106686"}
Nov 03 07:48:04.098909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:04.098852    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70a14375e83e21a7116b90e8cdc1ab72359e21c3acb77bd84615728cfa106686"
Nov 03 07:48:04.759003 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:48:04.758969    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:48:11.945897 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:11.945860    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"]
Nov 03 07:48:11.946338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:11.946311    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c670d8f1-5cc5-4519-b051-c7faeaeabb37" containerName="revoke-keys"
Nov 03 07:48:11.946338 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:11.946322    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c670d8f1-5cc5-4519-b051-c7faeaeabb37" containerName="revoke-keys"
Nov 03 07:48:11.946418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:11.946351    4470 cpu_manager.go:414] "containerMap: removing container" podUID="602e922a-3f3e-4388-a0f5-00d70b163d24" containerName="revoke-keys"
Nov 03 07:48:11.946418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:11.946357    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602e922a-3f3e-4388-a0f5-00d70b163d24" containerName="revoke-keys"
Nov 03 07:48:11.946483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:11.946425    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c670d8f1-5cc5-4519-b051-c7faeaeabb37" containerName="revoke-keys"
Nov 03 07:48:11.946483 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:11.946439    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="602e922a-3f3e-4388-a0f5-00d70b163d24" containerName="revoke-keys"
Nov 03 07:48:11.949499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:11.949482    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:12.056163 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.056099    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:48:12.056333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.056168    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:48:12.056333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.056170    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:48:12.056333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.056226    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:48:12.058892 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.058866    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"]
Nov 03 07:48:12.121300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.121269    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzrlh\" (UniqueName: \"kubernetes.io/projected/f37727eb-f06c-47fd-9a88-b6899a7505f5-kube-api-access-rzrlh\") pod \"maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:12.121490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.121327    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:12.121490 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.121475    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:12.223090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.223011    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:12.223283 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.223089    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzrlh\" (UniqueName: \"kubernetes.io/projected/f37727eb-f06c-47fd-9a88-b6899a7505f5-kube-api-access-rzrlh\") pod \"maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:12.223283 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.223158    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:12.223717 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.223690    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:12.225629 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.225605    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:12.287444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.287418    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzrlh\" (UniqueName: \"kubernetes.io/projected/f37727eb-f06c-47fd-9a88-b6899a7505f5-kube-api-access-rzrlh\") pod \"maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:12.562276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.562188    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:12.941393 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:12.941307    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"]
Nov 03 07:48:12.946454 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:48:12.946427    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37727eb_f06c_47fd_9a88_b6899a7505f5.slice/crio-925231f152b9c01c4b91d26db0c77a3282b8e79995e36ea17d340f1cccbfaa86 WatchSource:0}: Error finding container 925231f152b9c01c4b91d26db0c77a3282b8e79995e36ea17d340f1cccbfaa86: Status 404 returned error can't find the container with id 925231f152b9c01c4b91d26db0c77a3282b8e79995e36ea17d340f1cccbfaa86
Nov 03 07:48:13.153652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:13.153620    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9" event={"ID":"f37727eb-f06c-47fd-9a88-b6899a7505f5","Type":"ContainerStarted","Data":"925231f152b9c01c4b91d26db0c77a3282b8e79995e36ea17d340f1cccbfaa86"}
Nov 03 07:48:14.161090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:14.161004    4470 generic.go:356] "Generic (PLEG): container finished" podID="f37727eb-f06c-47fd-9a88-b6899a7505f5" containerID="1e2294f8d2be32ee7bb0248ddacc0a3736ba5765fbd68e0085af37fce6594711" exitCode=0
Nov 03 07:48:14.161491 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:14.161100    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9" event={"ID":"f37727eb-f06c-47fd-9a88-b6899a7505f5","Type":"ContainerDied","Data":"1e2294f8d2be32ee7bb0248ddacc0a3736ba5765fbd68e0085af37fce6594711"}
Nov 03 07:48:14.772195 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:14.772162    4470 scope.go:122] "RemoveContainer" containerID="36ab4483aee3cafdf2c3eb664fddcf608a6877bb5f3d099857843105e9fbb912"
Nov 03 07:48:14.781552 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:14.781529    4470 scope.go:122] "RemoveContainer" containerID="9736ada0b8b121b8fa43a34a0290c07a83141fc331f79526af250861f52fc577"
Nov 03 07:48:14.790415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:14.790396    4470 scope.go:122] "RemoveContainer" containerID="bc660400b7f3dfa465fab2ac50dc541ceb4b159c616863119db2f5772677c997"
Nov 03 07:48:14.798681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:14.798661    4470 scope.go:122] "RemoveContainer" containerID="7dd052153be9780ad848e86eabe6951f7639c73594b7203a25711785b9bbad9d"
Nov 03 07:48:14.807248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:14.807222    4470 scope.go:122] "RemoveContainer" containerID="c1651dc18d818820bfdbd3e20ef46661bb4108a2791155cde473369ced9696e4"
Nov 03 07:48:15.326096 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:15.326073    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:15.461618 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:15.461528    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f37727eb-f06c-47fd-9a88-b6899a7505f5-kube-api-access-rzrlh\" (UniqueName: \"kubernetes.io/projected/f37727eb-f06c-47fd-9a88-b6899a7505f5-kube-api-access-rzrlh\") pod \"f37727eb-f06c-47fd-9a88-b6899a7505f5\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") "
Nov 03 07:48:15.461795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:15.461693    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-xks-ca\") pod \"f37727eb-f06c-47fd-9a88-b6899a7505f5\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") "
Nov 03 07:48:15.461795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:15.461779    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-openshift-ca\") pod \"f37727eb-f06c-47fd-9a88-b6899a7505f5\" (UID: \"f37727eb-f06c-47fd-9a88-b6899a7505f5\") "
Nov 03 07:48:15.462185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:15.462126    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-openshift-ca" pod "f37727eb-f06c-47fd-9a88-b6899a7505f5" (UID: "f37727eb-f06c-47fd-9a88-b6899a7505f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:48:15.464408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:15.464376    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f37727eb-f06c-47fd-9a88-b6899a7505f5-kube-api-access-rzrlh" pod "f37727eb-f06c-47fd-9a88-b6899a7505f5" (UID: "f37727eb-f06c-47fd-9a88-b6899a7505f5"). InnerVolumeSpecName "kube-api-access-rzrlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:48:15.464512 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:15.464449    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-xks-ca" pod "f37727eb-f06c-47fd-9a88-b6899a7505f5" (UID: "f37727eb-f06c-47fd-9a88-b6899a7505f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:48:15.563418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:15.563382    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzrlh\" (UniqueName: \"kubernetes.io/projected/f37727eb-f06c-47fd-9a88-b6899a7505f5-kube-api-access-rzrlh\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:15.563418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:15.563413    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:15.563640 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:15.563424    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37727eb-f06c-47fd-9a88-b6899a7505f5-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:16.173843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:16.173765    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"
Nov 03 07:48:16.173843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:16.173797    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9" event={"ID":"f37727eb-f06c-47fd-9a88-b6899a7505f5","Type":"ContainerDied","Data":"925231f152b9c01c4b91d26db0c77a3282b8e79995e36ea17d340f1cccbfaa86"}
Nov 03 07:48:16.173843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:16.173843    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="925231f152b9c01c4b91d26db0c77a3282b8e79995e36ea17d340f1cccbfaa86"
Nov 03 07:48:19.724203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:19.724167    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"]
Nov 03 07:48:19.774119 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:19.774086    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6606d6-l5grv"]
Nov 03 07:48:19.805192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:19.805161    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7750ebd0-34bd-4b16-a462-3e43d6464421" path="/var/lib/kubelet/pods/7750ebd0-34bd-4b16-a462-3e43d6464421/volumes"
Nov 03 07:48:23.488943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:23.488909    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"]
Nov 03 07:48:23.596466 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:23.596433    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"]
Nov 03 07:48:27.753950 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:27.753916    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"]
Nov 03 07:48:27.754507 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:27.754116    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy" containerID="cri-o://c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214" gracePeriod=30
Nov 03 07:48:28.489409 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:28.489315    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n" podUID="7832737d-2343-4dfa-ba59-531727ea3ec1" containerName="payload-processing" containerID="cri-o://f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016" gracePeriod=25
Nov 03 07:48:28.597503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:28.597447    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh" podUID="841d918d-896d-4796-a0f6-546daac3f6bd" containerName="payload-pre-processing" containerID="cri-o://4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317" gracePeriod=25
Nov 03 07:48:28.691200 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:28.691161    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"]
Nov 03 07:48:28.741581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:28.741492    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eac348-v7mm9"]
Nov 03 07:48:29.805965 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:29.805933    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f37727eb-f06c-47fd-9a88-b6899a7505f5" path="/var/lib/kubelet/pods/f37727eb-f06c-47fd-9a88-b6899a7505f5/volumes"
Nov 03 07:48:33.002505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.002477    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:48:33.037686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.037605    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-socket\") pod \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") "
Nov 03 07:48:33.037686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.037643    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istiod-ca-cert\") pod \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") "
Nov 03 07:48:33.037923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.037713    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-token\" (UniqueName: \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-token\") pod \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") "
Nov 03 07:48:33.037991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.037918    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-socket" pod "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" (UID: "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:33.037991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.037928    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-podinfo\") pod \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") "
Nov 03 07:48:33.038108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038056    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-certs\") pod \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") "
Nov 03 07:48:33.038108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038094    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-envoy\") pod \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") "
Nov 03 07:48:33.038261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038119    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istiod-ca-cert" pod "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" (UID: "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:48:33.038261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038182    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-kube-api-access-7npp9\" (UniqueName: \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-kube-api-access-7npp9\") pod \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") "
Nov 03 07:48:33.038261 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038218    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-data\") pod \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") "
Nov 03 07:48:33.038417 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038278    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-credential-socket\") pod \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\" (UID: \"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9\") "
Nov 03 07:48:33.038417 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038326    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-certs" pod "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" (UID: "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:33.038681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038633    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:33.038681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038657    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:33.038681 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038662    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-credential-socket" pod "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" (UID: "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:33.038822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038672    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:33.038822 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.038700    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-data" pod "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" (UID: "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:33.040169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.040128    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-podinfo" pod "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" (UID: "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 07:48:33.040468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.040448    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-kube-api-access-7npp9" pod "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" (UID: "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9"). InnerVolumeSpecName "kube-api-access-7npp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:48:33.040662 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.040642    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-token" pod "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" (UID: "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:48:33.040761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.040672    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-envoy" pod "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" (UID: "d1e7eea4-e59f-48ff-9b67-07c640e0d0c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:33.139852 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.139821    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:33.139852 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.139848    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:33.139852 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.139858    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:33.140103 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.139868    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7npp9\" (UniqueName: \"kubernetes.io/projected/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-kube-api-access-7npp9\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:33.140103 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.139878    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:33.140103 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.139889    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:33.277778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.277741    4470 generic.go:356] "Generic (PLEG): container finished" podID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerID="c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214" exitCode=0
Nov 03 07:48:33.277957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.277794    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"
Nov 03 07:48:33.277957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.277823    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" event={"ID":"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9","Type":"ContainerDied","Data":"c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214"}
Nov 03 07:48:33.277957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.277881    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n" event={"ID":"d1e7eea4-e59f-48ff-9b67-07c640e0d0c9","Type":"ContainerDied","Data":"27279d36cb1614dbb7ba1ddcc1c57845e4c2526bd4a66c209f527bf702c73068"}
Nov 03 07:48:33.277957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.277906    4470 scope.go:122] "RemoveContainer" containerID="c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214"
Nov 03 07:48:33.288991 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.288970    4470 scope.go:122] "RemoveContainer" containerID="c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214"
Nov 03 07:48:33.289284 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:48:33.289265    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214\": container with ID starting with c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214 not found: ID does not exist" containerID="c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214"
Nov 03 07:48:33.289352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.289294    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214"} err="failed to get container status \"c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214\": rpc error: code = NotFound desc = could not find container \"c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214\": container with ID starting with c255813fd3c8d35151b48fd46f3317f7522a9122914cd053c594c73e76f5e214 not found: ID does not exist"
Nov 03 07:48:33.452256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.452226    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"]
Nov 03 07:48:33.458288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.458265    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n"]
Nov 03 07:48:33.805689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:33.805659    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" path="/var/lib/kubelet/pods/d1e7eea4-e59f-48ff-9b67-07c640e0d0c9/volumes"
Nov 03 07:48:34.759751 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:48:34.759709    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:48:37.833444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:37.833406    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh" podUID="841d918d-896d-4796-a0f6-546daac3f6bd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:48:37.833849 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:37.833420    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n" podUID="7832737d-2343-4dfa-ba59-531727ea3ec1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:48:42.077721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:42.077675    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"]
Nov 03 07:48:42.128430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:42.128400    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"]
Nov 03 07:48:47.833327 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:47.833283    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh" podUID="841d918d-896d-4796-a0f6-546daac3f6bd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:48:47.833728 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:47.833323    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n" podUID="7832737d-2343-4dfa-ba59-531727ea3ec1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:48:48.594463 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:48.594430    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"]
Nov 03 07:48:48.752110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:48.752050    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:48:48.848696 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:48.848675    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:48:48.911388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:48.911340    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7832737d-2343-4dfa-ba59-531727ea3ec1-kube-api-access-zqx9m\" (UniqueName: \"kubernetes.io/projected/7832737d-2343-4dfa-ba59-531727ea3ec1-kube-api-access-zqx9m\") pod \"7832737d-2343-4dfa-ba59-531727ea3ec1\" (UID: \"7832737d-2343-4dfa-ba59-531727ea3ec1\") "
Nov 03 07:48:48.911561 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:48.911410    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7832737d-2343-4dfa-ba59-531727ea3ec1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7832737d-2343-4dfa-ba59-531727ea3ec1-plugins-config-volume\") pod \"7832737d-2343-4dfa-ba59-531727ea3ec1\" (UID: \"7832737d-2343-4dfa-ba59-531727ea3ec1\") "
Nov 03 07:48:48.912097 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:48.912069    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7832737d-2343-4dfa-ba59-531727ea3ec1-plugins-config-volume" pod "7832737d-2343-4dfa-ba59-531727ea3ec1" (UID: "7832737d-2343-4dfa-ba59-531727ea3ec1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:48:48.914263 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:48.914236    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7832737d-2343-4dfa-ba59-531727ea3ec1-kube-api-access-zqx9m" pod "7832737d-2343-4dfa-ba59-531727ea3ec1" (UID: "7832737d-2343-4dfa-ba59-531727ea3ec1"). InnerVolumeSpecName "kube-api-access-zqx9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:48:49.017676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.017584    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841d918d-896d-4796-a0f6-546daac3f6bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841d918d-896d-4796-a0f6-546daac3f6bd-plugins-config-volume\") pod \"841d918d-896d-4796-a0f6-546daac3f6bd\" (UID: \"841d918d-896d-4796-a0f6-546daac3f6bd\") "
Nov 03 07:48:49.017676 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.017666    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841d918d-896d-4796-a0f6-546daac3f6bd-kube-api-access-rxnbt\" (UniqueName: \"kubernetes.io/projected/841d918d-896d-4796-a0f6-546daac3f6bd-kube-api-access-rxnbt\") pod \"841d918d-896d-4796-a0f6-546daac3f6bd\" (UID: \"841d918d-896d-4796-a0f6-546daac3f6bd\") "
Nov 03 07:48:49.018107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.018022    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqx9m\" (UniqueName: \"kubernetes.io/projected/7832737d-2343-4dfa-ba59-531727ea3ec1-kube-api-access-zqx9m\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:49.018295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.018113    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7832737d-2343-4dfa-ba59-531727ea3ec1-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:49.018633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.018577    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841d918d-896d-4796-a0f6-546daac3f6bd-plugins-config-volume" pod "841d918d-896d-4796-a0f6-546daac3f6bd" (UID: "841d918d-896d-4796-a0f6-546daac3f6bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:48:49.028171 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.024409    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841d918d-896d-4796-a0f6-546daac3f6bd-kube-api-access-rxnbt" pod "841d918d-896d-4796-a0f6-546daac3f6bd" (UID: "841d918d-896d-4796-a0f6-546daac3f6bd"). InnerVolumeSpecName "kube-api-access-rxnbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:48:49.118945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.118902    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841d918d-896d-4796-a0f6-546daac3f6bd-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:49.118945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.118938    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxnbt\" (UniqueName: \"kubernetes.io/projected/841d918d-896d-4796-a0f6-546daac3f6bd-kube-api-access-rxnbt\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:49.368306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.368272    4470 generic.go:356] "Generic (PLEG): container finished" podID="7832737d-2343-4dfa-ba59-531727ea3ec1" containerID="f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016" exitCode=0
Nov 03 07:48:49.368521 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.368319    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"
Nov 03 07:48:49.368521 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.368365    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n" event={"ID":"7832737d-2343-4dfa-ba59-531727ea3ec1","Type":"ContainerDied","Data":"f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016"}
Nov 03 07:48:49.368521 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.368423    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n" event={"ID":"7832737d-2343-4dfa-ba59-531727ea3ec1","Type":"ContainerDied","Data":"0b9b5a4e3c18d4a19cc6b20716021dd4f5e3e37e7c820936d47092c704e85c34"}
Nov 03 07:48:49.368521 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.368455    4470 scope.go:122] "RemoveContainer" containerID="f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016"
Nov 03 07:48:49.369972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.369948    4470 generic.go:356] "Generic (PLEG): container finished" podID="841d918d-896d-4796-a0f6-546daac3f6bd" containerID="4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317" exitCode=0
Nov 03 07:48:49.370286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.369991    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh" event={"ID":"841d918d-896d-4796-a0f6-546daac3f6bd","Type":"ContainerDied","Data":"4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317"}
Nov 03 07:48:49.370286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.370028    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"
Nov 03 07:48:49.370286 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.370030    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh" event={"ID":"841d918d-896d-4796-a0f6-546daac3f6bd","Type":"ContainerDied","Data":"a94d4a56ac7dd8ae59fe3a56667e4b8405f713dae1813bd317c127f9e51f1cf4"}
Nov 03 07:48:49.378316 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.378295    4470 scope.go:122] "RemoveContainer" containerID="f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016"
Nov 03 07:48:49.378629 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:48:49.378605    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016\": container with ID starting with f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016 not found: ID does not exist" containerID="f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016"
Nov 03 07:48:49.378720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.378640    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016"} err="failed to get container status \"f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016\": rpc error: code = NotFound desc = could not find container \"f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016\": container with ID starting with f9755bc24bea5973604f2e50636e0bba1382972d70f59ff98436d4184ffa0016 not found: ID does not exist"
Nov 03 07:48:49.378720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.378667    4470 scope.go:122] "RemoveContainer" containerID="4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317"
Nov 03 07:48:49.387414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.387398    4470 scope.go:122] "RemoveContainer" containerID="4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317"
Nov 03 07:48:49.387694 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:48:49.387673    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317\": container with ID starting with 4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317 not found: ID does not exist" containerID="4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317"
Nov 03 07:48:49.387775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.387704    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317"} err="failed to get container status \"4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317\": rpc error: code = NotFound desc = could not find container \"4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317\": container with ID starting with 4d36694803bf93580de8881138da054b277d27d26c312925dc8439260b13b317 not found: ID does not exist"
Nov 03 07:48:49.540281 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.540248    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"]
Nov 03 07:48:49.548657 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.548633    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh"]
Nov 03 07:48:49.662235 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.662157    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"]
Nov 03 07:48:49.667542 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.667513    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n"]
Nov 03 07:48:49.805179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.805151    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7832737d-2343-4dfa-ba59-531727ea3ec1" path="/var/lib/kubelet/pods/7832737d-2343-4dfa-ba59-531727ea3ec1/volumes"
Nov 03 07:48:49.805553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:49.805541    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841d918d-896d-4796-a0f6-546daac3f6bd" path="/var/lib/kubelet/pods/841d918d-896d-4796-a0f6-546daac3f6bd/volumes"
Nov 03 07:48:53.595435 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:53.595393    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp" podUID="dd717e60-b059-4bd2-9ca5-386a081b0f5d" containerName="payload-processing" containerID="cri-o://5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944" gracePeriod=25
Nov 03 07:48:53.852094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:53.852037    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:48:53.869249 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:53.869221    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd717e60-b059-4bd2-9ca5-386a081b0f5d-kube-api-access-5tbm2\" (UniqueName: \"kubernetes.io/projected/dd717e60-b059-4bd2-9ca5-386a081b0f5d-kube-api-access-5tbm2\") pod \"dd717e60-b059-4bd2-9ca5-386a081b0f5d\" (UID: \"dd717e60-b059-4bd2-9ca5-386a081b0f5d\") "
Nov 03 07:48:53.869379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:53.869300    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd717e60-b059-4bd2-9ca5-386a081b0f5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd717e60-b059-4bd2-9ca5-386a081b0f5d-plugins-config-volume\") pod \"dd717e60-b059-4bd2-9ca5-386a081b0f5d\" (UID: \"dd717e60-b059-4bd2-9ca5-386a081b0f5d\") "
Nov 03 07:48:53.869650 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:53.869628    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd717e60-b059-4bd2-9ca5-386a081b0f5d-plugins-config-volume" pod "dd717e60-b059-4bd2-9ca5-386a081b0f5d" (UID: "dd717e60-b059-4bd2-9ca5-386a081b0f5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:48:53.871383 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:53.871361    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd717e60-b059-4bd2-9ca5-386a081b0f5d-kube-api-access-5tbm2" pod "dd717e60-b059-4bd2-9ca5-386a081b0f5d" (UID: "dd717e60-b059-4bd2-9ca5-386a081b0f5d"). InnerVolumeSpecName "kube-api-access-5tbm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:48:53.970404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:53.970370    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tbm2\" (UniqueName: \"kubernetes.io/projected/dd717e60-b059-4bd2-9ca5-386a081b0f5d-kube-api-access-5tbm2\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:53.970404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:53.970400    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd717e60-b059-4bd2-9ca5-386a081b0f5d-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:54.403670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:54.403640    4470 generic.go:356] "Generic (PLEG): container finished" podID="dd717e60-b059-4bd2-9ca5-386a081b0f5d" containerID="5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944" exitCode=0
Nov 03 07:48:54.403868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:54.403687    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"
Nov 03 07:48:54.403868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:54.403736    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp" event={"ID":"dd717e60-b059-4bd2-9ca5-386a081b0f5d","Type":"ContainerDied","Data":"5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944"}
Nov 03 07:48:54.403868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:54.403790    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp" event={"ID":"dd717e60-b059-4bd2-9ca5-386a081b0f5d","Type":"ContainerDied","Data":"e79f5dece03abaeae210127e51245b3ea1d2b08236ca08622cbdda29ba5cf49a"}
Nov 03 07:48:54.403868 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:54.403815    4470 scope.go:122] "RemoveContainer" containerID="5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944"
Nov 03 07:48:54.413203 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:54.413185    4470 scope.go:122] "RemoveContainer" containerID="5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944"
Nov 03 07:48:54.413505 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:48:54.413485    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944\": container with ID starting with 5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944 not found: ID does not exist" containerID="5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944"
Nov 03 07:48:54.413553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:54.413514    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944"} err="failed to get container status \"5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944\": rpc error: code = NotFound desc = could not find container \"5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944\": container with ID starting with 5d04959043539fdd2697b20fa9b151ab066eef9e8762998b31d315da2fc41944 not found: ID does not exist"
Nov 03 07:48:54.575323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:54.575294    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"]
Nov 03 07:48:54.580812 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:54.580779    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp"]
Nov 03 07:48:55.805654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:55.805623    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd717e60-b059-4bd2-9ca5-386a081b0f5d" path="/var/lib/kubelet/pods/dd717e60-b059-4bd2-9ca5-386a081b0f5d/volumes"
Nov 03 07:48:56.332802 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.332768    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"]
Nov 03 07:48:56.333297 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333277    4470 cpu_manager.go:414] "containerMap: removing container" podUID="dd717e60-b059-4bd2-9ca5-386a081b0f5d" containerName="payload-processing"
Nov 03 07:48:56.333418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333298    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd717e60-b059-4bd2-9ca5-386a081b0f5d" containerName="payload-processing"
Nov 03 07:48:56.333418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333316    4470 cpu_manager.go:414] "containerMap: removing container" podUID="841d918d-896d-4796-a0f6-546daac3f6bd" containerName="payload-pre-processing"
Nov 03 07:48:56.333418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333324    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841d918d-896d-4796-a0f6-546daac3f6bd" containerName="payload-pre-processing"
Nov 03 07:48:56.333418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333346    4470 cpu_manager.go:414] "containerMap: removing container" podUID="f37727eb-f06c-47fd-9a88-b6899a7505f5" containerName="revoke-keys"
Nov 03 07:48:56.333418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333353    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f37727eb-f06c-47fd-9a88-b6899a7505f5" containerName="revoke-keys"
Nov 03 07:48:56.333418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333371    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy"
Nov 03 07:48:56.333418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333378    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy"
Nov 03 07:48:56.333418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333391    4470 cpu_manager.go:414] "containerMap: removing container" podUID="7832737d-2343-4dfa-ba59-531727ea3ec1" containerName="payload-processing"
Nov 03 07:48:56.333418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333398    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7832737d-2343-4dfa-ba59-531727ea3ec1" containerName="payload-processing"
Nov 03 07:48:56.333881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333501    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="841d918d-896d-4796-a0f6-546daac3f6bd" containerName="payload-pre-processing"
Nov 03 07:48:56.333881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333517    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="f37727eb-f06c-47fd-9a88-b6899a7505f5" containerName="revoke-keys"
Nov 03 07:48:56.333881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333530    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e7eea4-e59f-48ff-9b67-07c640e0d0c9" containerName="istio-proxy"
Nov 03 07:48:56.333881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333540    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="7832737d-2343-4dfa-ba59-531727ea3ec1" containerName="payload-processing"
Nov 03 07:48:56.333881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.333556    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd717e60-b059-4bd2-9ca5-386a081b0f5d" containerName="payload-processing"
Nov 03 07:48:56.337972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.337952    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:56.390499 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.390458    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0798fa-bpw6j\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:56.390805 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.390779    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0798fa-bpw6j\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:56.390923 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.390824    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqgdj\" (UniqueName: \"kubernetes.io/projected/e3e765df-1366-4bf0-8b4f-38bed6cfe159-kube-api-access-qqgdj\") pod \"maas-api-revoke-sub-bf0798fa-bpw6j\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:56.397210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.397176    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"]
Nov 03 07:48:56.444265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.444218    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:48:56.444265 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.444259    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:48:56.444497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.444273    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:48:56.444497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.444264    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:48:56.491998 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.491966    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0798fa-bpw6j\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:56.492199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.492010    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0798fa-bpw6j\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:56.492288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.492214    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqgdj\" (UniqueName: \"kubernetes.io/projected/e3e765df-1366-4bf0-8b4f-38bed6cfe159-kube-api-access-qqgdj\") pod \"maas-api-revoke-sub-bf0798fa-bpw6j\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:56.492785 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.492758    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0798fa-bpw6j\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:56.494626 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.494607    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0798fa-bpw6j\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:56.556846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.556808    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqgdj\" (UniqueName: \"kubernetes.io/projected/e3e765df-1366-4bf0-8b4f-38bed6cfe159-kube-api-access-qqgdj\") pod \"maas-api-revoke-sub-bf0798fa-bpw6j\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:56.651025 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:56.650942    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:57.025875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.025849    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"]
Nov 03 07:48:57.027330 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:48:57.027304    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e765df_1366_4bf0_8b4f_38bed6cfe159.slice/crio-470bcf816b9732a9975d658406d4f8f9a08b1bae4931d61c2a7a190858dd36bf WatchSource:0}: Error finding container 470bcf816b9732a9975d658406d4f8f9a08b1bae4931d61c2a7a190858dd36bf: Status 404 returned error can't find the container with id 470bcf816b9732a9975d658406d4f8f9a08b1bae4931d61c2a7a190858dd36bf
Nov 03 07:48:57.098424 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.098378    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" containerID="cri-o://6631380b5bc3110c484187abc55abfcd2aac8c32eb4f58199eaff72d2a22d603" gracePeriod=45
Nov 03 07:48:57.149496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.149456    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main" containerID="cri-o://f51019aa959d32b857d7062165266b1ac175caf25f86e8eb60f103544f26de23" gracePeriod=45
Nov 03 07:48:57.216216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.216180    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": dial tcp 10.134.0.89:8000: connect: connection refused"
Nov 03 07:48:57.427182 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.427117    4470 generic.go:356] "Generic (PLEG): container finished" podID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerID="f51019aa959d32b857d7062165266b1ac175caf25f86e8eb60f103544f26de23" exitCode=0
Nov 03 07:48:57.427428 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.427297    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" event={"ID":"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4","Type":"ContainerDied","Data":"f51019aa959d32b857d7062165266b1ac175caf25f86e8eb60f103544f26de23"}
Nov 03 07:48:57.429918 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.429889    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j" event={"ID":"e3e765df-1366-4bf0-8b4f-38bed6cfe159","Type":"ContainerStarted","Data":"fbf3ffbe9301b0b054f757c9854eff5032d9c5efc446a5f213a14b718b0804e2"}
Nov 03 07:48:57.430066 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.429986    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j" event={"ID":"e3e765df-1366-4bf0-8b4f-38bed6cfe159","Type":"ContainerStarted","Data":"470bcf816b9732a9975d658406d4f8f9a08b1bae4931d61c2a7a190858dd36bf"}
Nov 03 07:48:57.431709 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.431683    4470 generic.go:356] "Generic (PLEG): container finished" podID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerID="6631380b5bc3110c484187abc55abfcd2aac8c32eb4f58199eaff72d2a22d603" exitCode=0
Nov 03 07:48:57.431829 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.431727    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" event={"ID":"a1ba71f7-2c52-49e3-845d-6f89f1d9157a","Type":"ContainerDied","Data":"6631380b5bc3110c484187abc55abfcd2aac8c32eb4f58199eaff72d2a22d603"}
Nov 03 07:48:57.442675 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.442653    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:48:57.453586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.453563    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:48:57.502371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502220    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-dshm\") pod \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") "
Nov 03 07:48:57.502371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502296    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-model-cache\") pod \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") "
Nov 03 07:48:57.502593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502389    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-kube-api-access-qc62r\" (UniqueName: \"kubernetes.io/projected/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-kube-api-access-qc62r\") pod \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") "
Nov 03 07:48:57.502593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502422    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tls-certs\" (UniqueName: \"kubernetes.io/secret/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tls-certs\") pod \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") "
Nov 03 07:48:57.502593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502457    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-kube-api-access-bbqzt\" (UniqueName: \"kubernetes.io/projected/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-kube-api-access-bbqzt\") pod \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") "
Nov 03 07:48:57.502593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502489    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-home\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-home\") pod \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") "
Nov 03 07:48:57.502593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502523    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-model-cache\") pod \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") "
Nov 03 07:48:57.502593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502539    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-model-cache" pod "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" (UID: "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:57.502593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502562    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tls-certs\") pod \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") "
Nov 03 07:48:57.503093 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502627    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-home\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-home\") pod \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") "
Nov 03 07:48:57.503093 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502674    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tmp-dir\") pod \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\" (UID: \"a1ba71f7-2c52-49e3-845d-6f89f1d9157a\") "
Nov 03 07:48:57.503093 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502746    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tmp-dir\") pod \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") "
Nov 03 07:48:57.503093 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.502788    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-dshm\") pod \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\" (UID: \"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4\") "
Nov 03 07:48:57.503598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.503344    4470 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-model-cache\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.503683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.503647    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-home" pod "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" (UID: "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:57.504842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.503943    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tmp-dir" pod "a1ba71f7-2c52-49e3-845d-6f89f1d9157a" (UID: "a1ba71f7-2c52-49e3-845d-6f89f1d9157a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:57.504842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.504219    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-home" pod "a1ba71f7-2c52-49e3-845d-6f89f1d9157a" (UID: "a1ba71f7-2c52-49e3-845d-6f89f1d9157a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:57.504842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.504224    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-model-cache" pod "a1ba71f7-2c52-49e3-845d-6f89f1d9157a" (UID: "a1ba71f7-2c52-49e3-845d-6f89f1d9157a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:57.504842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.504261    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tmp-dir" pod "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" (UID: "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:57.505659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.505623    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-dshm" pod "a1ba71f7-2c52-49e3-845d-6f89f1d9157a" (UID: "a1ba71f7-2c52-49e3-845d-6f89f1d9157a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:57.505659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.505637    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tls-certs" pod "a1ba71f7-2c52-49e3-845d-6f89f1d9157a" (UID: "a1ba71f7-2c52-49e3-845d-6f89f1d9157a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:48:57.506071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.506047    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-kube-api-access-qc62r" pod "a1ba71f7-2c52-49e3-845d-6f89f1d9157a" (UID: "a1ba71f7-2c52-49e3-845d-6f89f1d9157a"). InnerVolumeSpecName "kube-api-access-qc62r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:48:57.507379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.507357    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-dshm" pod "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" (UID: "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:48:57.507763 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.507744    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-kube-api-access-bbqzt" pod "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" (UID: "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4"). InnerVolumeSpecName "kube-api-access-bbqzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:48:57.507952 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.507932    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tls-certs" pod "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" (UID: "0db08b8e-2684-4a15-8eee-2bd8d07c9ee4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:48:57.603956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.603855    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbqzt\" (UniqueName: \"kubernetes.io/projected/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-kube-api-access-bbqzt\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.603956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.603892    4470 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-home\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.603956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.603906    4470 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-model-cache\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.603956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.603919    4470 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tls-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.603956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.603933    4470 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-home\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.603956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.603944    4470 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-tmp-dir\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.603956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.603955    4470 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tmp-dir\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.604495 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.603966    4470 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-dshm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.604495 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.603979    4470 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-dshm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.604495 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.603989    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc62r\" (UniqueName: \"kubernetes.io/projected/a1ba71f7-2c52-49e3-845d-6f89f1d9157a-kube-api-access-qc62r\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.604495 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.604000    4470 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4-tls-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:57.673094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:57.672889    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j" podStartSLOduration=1.672868378 podStartE2EDuration="1.672868378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:48:57.550379546 +0000 UTC m=+2989.882748541" watchObservedRunningTime="2026-11-03 07:48:57.672868378 +0000 UTC m=+2990.005237373"
Nov 03 07:48:58.438788 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.438753    4470 generic.go:356] "Generic (PLEG): container finished" podID="e3e765df-1366-4bf0-8b4f-38bed6cfe159" containerID="fbf3ffbe9301b0b054f757c9854eff5032d9c5efc446a5f213a14b718b0804e2" exitCode=0
Nov 03 07:48:58.439278 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.438843    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j" event={"ID":"e3e765df-1366-4bf0-8b4f-38bed6cfe159","Type":"ContainerDied","Data":"fbf3ffbe9301b0b054f757c9854eff5032d9c5efc446a5f213a14b718b0804e2"}
Nov 03 07:48:58.440237 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.440215    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"
Nov 03 07:48:58.440395 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.440268    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz" event={"ID":"a1ba71f7-2c52-49e3-845d-6f89f1d9157a","Type":"ContainerDied","Data":"07ff8fc2fcb60909c641e9f9efe4cb62c01ff6018f3e6cd2632e10a32ed96d55"}
Nov 03 07:48:58.440395 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.440311    4470 scope.go:122] "RemoveContainer" containerID="6631380b5bc3110c484187abc55abfcd2aac8c32eb4f58199eaff72d2a22d603"
Nov 03 07:48:58.441827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.441807    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f" event={"ID":"0db08b8e-2684-4a15-8eee-2bd8d07c9ee4","Type":"ContainerDied","Data":"88be6709f740ff8fdc4049ef4b9f6336a2e5354e74862092649aa34e4a237dae"}
Nov 03 07:48:58.441827 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.441820    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"
Nov 03 07:48:58.450044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.450026    4470 scope.go:122] "RemoveContainer" containerID="f51019aa959d32b857d7062165266b1ac175caf25f86e8eb60f103544f26de23"
Nov 03 07:48:58.675166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.675115    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"]
Nov 03 07:48:58.684051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.684021    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-2-w4-ffee69-kserve-5bfb8c469-pmhnz"]
Nov 03 07:48:58.798291 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.798214    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"]
Nov 03 07:48:58.806980 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:58.806951    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/premium-w4-ffee69-kserve-6487b8fd79-9sc6f"]
Nov 03 07:48:59.589170 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.589148    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:48:59.622628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.622601    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-openshift-ca\") pod \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") "
Nov 03 07:48:59.622791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.622662    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e765df-1366-4bf0-8b4f-38bed6cfe159-kube-api-access-qqgdj\" (UniqueName: \"kubernetes.io/projected/e3e765df-1366-4bf0-8b4f-38bed6cfe159-kube-api-access-qqgdj\") pod \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") "
Nov 03 07:48:59.622791 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.622693    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-xks-ca\") pod \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\" (UID: \"e3e765df-1366-4bf0-8b4f-38bed6cfe159\") "
Nov 03 07:48:59.622988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.622967    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-openshift-ca" pod "e3e765df-1366-4bf0-8b4f-38bed6cfe159" (UID: "e3e765df-1366-4bf0-8b4f-38bed6cfe159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:48:59.624924 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.624902    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-xks-ca" pod "e3e765df-1366-4bf0-8b4f-38bed6cfe159" (UID: "e3e765df-1366-4bf0-8b4f-38bed6cfe159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:48:59.625000 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.624960    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e765df-1366-4bf0-8b4f-38bed6cfe159-kube-api-access-qqgdj" pod "e3e765df-1366-4bf0-8b4f-38bed6cfe159" (UID: "e3e765df-1366-4bf0-8b4f-38bed6cfe159"). InnerVolumeSpecName "kube-api-access-qqgdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:48:59.724022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.723945    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:59.724022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.723983    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqgdj\" (UniqueName: \"kubernetes.io/projected/e3e765df-1366-4bf0-8b4f-38bed6cfe159-kube-api-access-qqgdj\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:59.724022 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.724000    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e765df-1366-4bf0-8b4f-38bed6cfe159-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:48:59.805674 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.805635    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" path="/var/lib/kubelet/pods/0db08b8e-2684-4a15-8eee-2bd8d07c9ee4/volumes"
Nov 03 07:48:59.806167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:48:59.806118    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" path="/var/lib/kubelet/pods/a1ba71f7-2c52-49e3-845d-6f89f1d9157a/volumes"
Nov 03 07:49:00.457317 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:00.457286    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"
Nov 03 07:49:00.457497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:00.457319    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j" event={"ID":"e3e765df-1366-4bf0-8b4f-38bed6cfe159","Type":"ContainerDied","Data":"470bcf816b9732a9975d658406d4f8f9a08b1bae4931d61c2a7a190858dd36bf"}
Nov 03 07:49:00.457497 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:00.457366    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="470bcf816b9732a9975d658406d4f8f9a08b1bae4931d61c2a7a190858dd36bf"
Nov 03 07:49:04.758979 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:49:04.758937    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:49:34.827308 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:49:34.827227    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:49:50.981775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:50.981743    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"]
Nov 03 07:49:50.982210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:50.982147    4470 cpu_manager.go:414] "containerMap: removing container" podUID="e3e765df-1366-4bf0-8b4f-38bed6cfe159" containerName="revoke-keys"
Nov 03 07:49:50.982210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:50.982159    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e765df-1366-4bf0-8b4f-38bed6cfe159" containerName="revoke-keys"
Nov 03 07:49:50.982210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:50.982170    4470 cpu_manager.go:414] "containerMap: removing container" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main"
Nov 03 07:49:50.982210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:50.982175    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main"
Nov 03 07:49:50.982210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:50.982195    4470 cpu_manager.go:414] "containerMap: removing container" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main"
Nov 03 07:49:50.982210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:50.982200    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main"
Nov 03 07:49:50.982419 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:50.982269    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db08b8e-2684-4a15-8eee-2bd8d07c9ee4" containerName="main"
Nov 03 07:49:50.982419 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:50.982282    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e765df-1366-4bf0-8b4f-38bed6cfe159" containerName="revoke-keys"
Nov 03 07:49:50.982419 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:50.982289    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ba71f7-2c52-49e3-845d-6f89f1d9157a" containerName="main"
Nov 03 07:49:50.985481 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:50.985463    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:51.044721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.044689    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"]
Nov 03 07:49:51.074788 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.074758    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ckphc\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:51.074964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.074830    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xdw\" (UniqueName: \"kubernetes.io/projected/52cec711-289e-4862-b94c-daa4d2fa0281-kube-api-access-s6xdw\") pod \"maas-api-revoke-sub-85237e45-ckphc\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:51.074964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.074940    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ckphc\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:51.092076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.092053    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:49:51.092268 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.092249    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:49:51.092368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.092350    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:49:51.092412 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.092364    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:49:51.176503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.176467    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xdw\" (UniqueName: \"kubernetes.io/projected/52cec711-289e-4862-b94c-daa4d2fa0281-kube-api-access-s6xdw\") pod \"maas-api-revoke-sub-85237e45-ckphc\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:51.176503 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.176508    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ckphc\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:51.176734 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.176562    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ckphc\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:51.177262 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.177243    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ckphc\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:51.179042 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.179020    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ckphc\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:51.243218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.243123    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xdw\" (UniqueName: \"kubernetes.io/projected/52cec711-289e-4862-b94c-daa4d2fa0281-kube-api-access-s6xdw\") pod \"maas-api-revoke-sub-85237e45-ckphc\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:51.298308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.298270    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:51.676694 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.676666    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"]
Nov 03 07:49:51.677754 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:49:51.677732    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52cec711_289e_4862_b94c_daa4d2fa0281.slice/crio-9e49b5f45937be2db423e4db2d1edb87fb6c4795d6f49349e4b9ad4b3dadcf02 WatchSource:0}: Error finding container 9e49b5f45937be2db423e4db2d1edb87fb6c4795d6f49349e4b9ad4b3dadcf02: Status 404 returned error can't find the container with id 9e49b5f45937be2db423e4db2d1edb87fb6c4795d6f49349e4b9ad4b3dadcf02
Nov 03 07:49:51.751365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:51.751331    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc" event={"ID":"52cec711-289e-4862-b94c-daa4d2fa0281","Type":"ContainerStarted","Data":"9e49b5f45937be2db423e4db2d1edb87fb6c4795d6f49349e4b9ad4b3dadcf02"}
Nov 03 07:49:52.758759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:52.758670    4470 generic.go:356] "Generic (PLEG): container finished" podID="52cec711-289e-4862-b94c-daa4d2fa0281" containerID="147bc04255287a6cc75f5ccb9b8320787eb07333c3e8f8420abea8f890030ab3" exitCode=0
Nov 03 07:49:52.758759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:52.758729    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc" event={"ID":"52cec711-289e-4862-b94c-daa4d2fa0281","Type":"ContainerDied","Data":"147bc04255287a6cc75f5ccb9b8320787eb07333c3e8f8420abea8f890030ab3"}
Nov 03 07:49:53.901961 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:53.901933    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:54.004638 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.004535    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-openshift-ca\") pod \"52cec711-289e-4862-b94c-daa4d2fa0281\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") "
Nov 03 07:49:54.004984 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.004954    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52cec711-289e-4862-b94c-daa4d2fa0281-kube-api-access-s6xdw\" (UniqueName: \"kubernetes.io/projected/52cec711-289e-4862-b94c-daa4d2fa0281-kube-api-access-s6xdw\") pod \"52cec711-289e-4862-b94c-daa4d2fa0281\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") "
Nov 03 07:49:54.005108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.005016    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-xks-ca\") pod \"52cec711-289e-4862-b94c-daa4d2fa0281\" (UID: \"52cec711-289e-4862-b94c-daa4d2fa0281\") "
Nov 03 07:49:54.005208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.004950    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-openshift-ca" pod "52cec711-289e-4862-b94c-daa4d2fa0281" (UID: "52cec711-289e-4862-b94c-daa4d2fa0281"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:49:54.005413 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.005398    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:49:54.007223 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.007199    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-xks-ca" pod "52cec711-289e-4862-b94c-daa4d2fa0281" (UID: "52cec711-289e-4862-b94c-daa4d2fa0281"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:49:54.007314 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.007264    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52cec711-289e-4862-b94c-daa4d2fa0281-kube-api-access-s6xdw" pod "52cec711-289e-4862-b94c-daa4d2fa0281" (UID: "52cec711-289e-4862-b94c-daa4d2fa0281"). InnerVolumeSpecName "kube-api-access-s6xdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:49:54.106006 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.105970    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xdw\" (UniqueName: \"kubernetes.io/projected/52cec711-289e-4862-b94c-daa4d2fa0281-kube-api-access-s6xdw\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:49:54.106006 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.106003    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52cec711-289e-4862-b94c-daa4d2fa0281-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:49:54.771467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.771436    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"
Nov 03 07:49:54.771467 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.771444    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc" event={"ID":"52cec711-289e-4862-b94c-daa4d2fa0281","Type":"ContainerDied","Data":"9e49b5f45937be2db423e4db2d1edb87fb6c4795d6f49349e4b9ad4b3dadcf02"}
Nov 03 07:49:54.771703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:54.771495    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e49b5f45937be2db423e4db2d1edb87fb6c4795d6f49349e4b9ad4b3dadcf02"
Nov 03 07:49:59.397893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.397853    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"]
Nov 03 07:49:59.398320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.398297    4470 cpu_manager.go:414] "containerMap: removing container" podUID="52cec711-289e-4862-b94c-daa4d2fa0281" containerName="revoke-keys"
Nov 03 07:49:59.398320 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.398314    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52cec711-289e-4862-b94c-daa4d2fa0281" containerName="revoke-keys"
Nov 03 07:49:59.398416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.398405    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="52cec711-289e-4862-b94c-daa4d2fa0281" containerName="revoke-keys"
Nov 03 07:49:59.401738 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.401719    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.509731 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.509696    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"]
Nov 03 07:49:59.559593 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.559554    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.559777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.559603    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.559777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.559702    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.559777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.559732    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.559960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.559872    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.559960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.559938    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.560073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.560025    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm57j\" (UniqueName: \"kubernetes.io/projected/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-kube-api-access-tm57j\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.560115 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.560099    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.560178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.560126    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.661662 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.661576    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.661662 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.661617    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.661662 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.661648    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.661930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.661715    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.661930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.661778    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.661930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.661800    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.661930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.661839    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.661930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.661891    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.662288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.661951    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm57j\" (UniqueName: \"kubernetes.io/projected/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-kube-api-access-tm57j\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.662288 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.662265    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.662412 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.662346    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.662530 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.662504    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.662652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.662513    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.662652 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.662610    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.664221 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.664200    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.664322 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.664304    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.728432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.728393    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:49:59.728651 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:49:59.728628    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm57j\" (UniqueName: \"kubernetes.io/projected/4dfa6c32-f24d-4d1f-96ff-df222dec2a7e-kube-api-access-tm57j\") pod \"maas-default-gateway-openshift-default-58d898bd55-7bv22\" (UID: \"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:50:00.014237 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:00.014148    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:50:00.447153 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:00.447106    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"]
Nov 03 07:50:00.448576 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:50:00.448548    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfa6c32_f24d_4d1f_96ff_df222dec2a7e.slice/crio-8e4b05280aa31ca6febf07cfe9a75a083e17329e60581fbd3571730c0f653c00 WatchSource:0}: Error finding container 8e4b05280aa31ca6febf07cfe9a75a083e17329e60581fbd3571730c0f653c00: Status 404 returned error can't find the container with id 8e4b05280aa31ca6febf07cfe9a75a083e17329e60581fbd3571730c0f653c00
Nov 03 07:50:00.450795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:00.450757    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:50:00.450909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:00.450844    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:50:00.450909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:00.450890    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 07:50:00.808981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:00.808885    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" event={"ID":"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e","Type":"ContainerStarted","Data":"92f2a043cbacd01be8ed6fe3a6291a03d4db4aa1bd1288c45cb8b2061d3fa722"}
Nov 03 07:50:00.808981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:00.808948    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" event={"ID":"4dfa6c32-f24d-4d1f-96ff-df222dec2a7e","Type":"ContainerStarted","Data":"8e4b05280aa31ca6febf07cfe9a75a083e17329e60581fbd3571730c0f653c00"}
Nov 03 07:50:00.934733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:00.934679    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" podStartSLOduration=1.9346612749999998 podStartE2EDuration="1.934661275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:50:00.934307227 +0000 UTC m=+3053.266676241" watchObservedRunningTime="2026-11-03 07:50:00.934661275 +0000 UTC m=+3053.267030269"
Nov 03 07:50:01.014362 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:01.014335    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:50:01.015680 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:01.015660    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 03 07:50:01.015759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:01.015702    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" podUID="4dfa6c32-f24d-4d1f-96ff-df222dec2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 03 07:50:02.014650 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:02.014609    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 03 07:50:02.015198 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:02.014683    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" podUID="4dfa6c32-f24d-4d1f-96ff-df222dec2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 03 07:50:03.015574 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:03.015532    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 03 07:50:03.016019 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:03.015608    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" podUID="4dfa6c32-f24d-4d1f-96ff-df222dec2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 03 07:50:04.014943 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:04.014908    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 03 07:50:04.015111 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:04.014970    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" podUID="4dfa6c32-f24d-4d1f-96ff-df222dec2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 03 07:50:04.780403 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:50:04.780366    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:50:05.015124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:05.015089    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 03 07:50:05.015124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:05.015152    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" podUID="4dfa6c32-f24d-4d1f-96ff-df222dec2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 03 07:50:06.015107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:06.015072    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 03 07:50:06.015654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:06.015166    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" podUID="4dfa6c32-f24d-4d1f-96ff-df222dec2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 03 07:50:07.016815 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:07.016714    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 03 07:50:07.017892 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:07.017404    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" podUID="4dfa6c32-f24d-4d1f-96ff-df222dec2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 03 07:50:08.706853 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:08.706817    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:50:08.707354 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:08.707232    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:50:08.707988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:08.707970    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22"
Nov 03 07:50:14.888754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:14.888721    4470 scope.go:122] "RemoveContainer" containerID="6a2ee6982e5262fec9b210e5be5ca1922d888e71edae2e84bfef583f9d8a1f67"
Nov 03 07:50:14.898049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:14.898025    4470 scope.go:122] "RemoveContainer" containerID="0dd6d06197f2fe83935575176b08459de13fbf325e70bb14dd31eaa28e516345"
Nov 03 07:50:30.057561 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:30.057526    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"]
Nov 03 07:50:30.108364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:30.108322    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d47bf7-dlqlr"]
Nov 03 07:50:31.805049 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:31.805019    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7702a8-fd0f-44f0-a293-6d9980c7fc73" path="/var/lib/kubelet/pods/7f7702a8-fd0f-44f0-a293-6d9980c7fc73/volumes"
Nov 03 07:50:34.758458 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:50:34.758413    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:50:43.060519 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:43.060485    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"]
Nov 03 07:50:43.110795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:43.110763    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4158a54e-m4tjv"]
Nov 03 07:50:43.805891 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:43.805857    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c425ee9-4286-4ab3-a737-c01ab814de0e" path="/var/lib/kubelet/pods/7c425ee9-4286-4ab3-a737-c01ab814de0e/volumes"
Nov 03 07:50:57.642431 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.642351    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"]
Nov 03 07:50:57.646074 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.646057    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:50:57.704896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.704848    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"]
Nov 03 07:50:57.737915 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.737874    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:50:57.752127 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.752096    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:50:57.752307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.752159    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:50:57.752307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.752239    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:50:57.771094 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.771057    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrkbr\" (UniqueName: \"kubernetes.io/projected/d4abaca6-d153-428d-8f2a-f2c1628046b9-kube-api-access-lrkbr\") pod \"maas-api-revoke-sub-c1e271a9-x7n92\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:50:57.771307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.771228    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-x7n92\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:50:57.771307 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.771284    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-x7n92\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:50:57.872641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.872598    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrkbr\" (UniqueName: \"kubernetes.io/projected/d4abaca6-d153-428d-8f2a-f2c1628046b9-kube-api-access-lrkbr\") pod \"maas-api-revoke-sub-c1e271a9-x7n92\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:50:57.872831 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.872678    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-x7n92\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:50:57.872831 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.872743    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-x7n92\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:50:57.873359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.873337    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-x7n92\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:50:57.875210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.875189    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-x7n92\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:50:57.936996 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.936923    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrkbr\" (UniqueName: \"kubernetes.io/projected/d4abaca6-d153-428d-8f2a-f2c1628046b9-kube-api-access-lrkbr\") pod \"maas-api-revoke-sub-c1e271a9-x7n92\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:50:57.959474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:57.959446    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:50:58.336348 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:58.336316    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"]
Nov 03 07:50:58.336991 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:50:58.336966    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4abaca6_d153_428d_8f2a_f2c1628046b9.slice/crio-26c5eb3d379810d9fbe365948878ba2c0e9264e00e8c445bc51368a07aaa30b1 WatchSource:0}: Error finding container 26c5eb3d379810d9fbe365948878ba2c0e9264e00e8c445bc51368a07aaa30b1: Status 404 returned error can't find the container with id 26c5eb3d379810d9fbe365948878ba2c0e9264e00e8c445bc51368a07aaa30b1
Nov 03 07:50:59.130601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:59.130566    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92" event={"ID":"d4abaca6-d153-428d-8f2a-f2c1628046b9","Type":"ContainerStarted","Data":"3be77d8f1df15a6ed67a10a6af10cb36c3efa98c077547d597bca7fcd6eb9d4f"}
Nov 03 07:50:59.131032 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:59.130629    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92" event={"ID":"d4abaca6-d153-428d-8f2a-f2c1628046b9","Type":"ContainerStarted","Data":"26c5eb3d379810d9fbe365948878ba2c0e9264e00e8c445bc51368a07aaa30b1"}
Nov 03 07:50:59.253716 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:50:59.253658    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92" podStartSLOduration=2.25364368 podStartE2EDuration="2.25364368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:50:59.251053124 +0000 UTC m=+3111.583422121" watchObservedRunningTime="2026-11-03 07:50:59.25364368 +0000 UTC m=+3111.586012674"
Nov 03 07:51:00.057883 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:00.057852    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"]
Nov 03 07:51:00.108361 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:00.108327    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"]
Nov 03 07:51:00.110341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:00.110326    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1208e6-wm4tf"]
Nov 03 07:51:00.112073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:00.112050    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c41d424-54pxw"]
Nov 03 07:51:00.141252 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:00.141214    4470 generic.go:356] "Generic (PLEG): container finished" podID="d4abaca6-d153-428d-8f2a-f2c1628046b9" containerID="3be77d8f1df15a6ed67a10a6af10cb36c3efa98c077547d597bca7fcd6eb9d4f" exitCode=0
Nov 03 07:51:00.141708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:00.141287    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92" event={"ID":"d4abaca6-d153-428d-8f2a-f2c1628046b9","Type":"ContainerDied","Data":"3be77d8f1df15a6ed67a10a6af10cb36c3efa98c077547d597bca7fcd6eb9d4f"}
Nov 03 07:51:01.280896 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.280868    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:51:01.409046 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.408962    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-openshift-ca\") pod \"d4abaca6-d153-428d-8f2a-f2c1628046b9\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") "
Nov 03 07:51:01.409046 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.409014    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-xks-ca\") pod \"d4abaca6-d153-428d-8f2a-f2c1628046b9\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") "
Nov 03 07:51:01.409315 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.409092    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4abaca6-d153-428d-8f2a-f2c1628046b9-kube-api-access-lrkbr\" (UniqueName: \"kubernetes.io/projected/d4abaca6-d153-428d-8f2a-f2c1628046b9-kube-api-access-lrkbr\") pod \"d4abaca6-d153-428d-8f2a-f2c1628046b9\" (UID: \"d4abaca6-d153-428d-8f2a-f2c1628046b9\") "
Nov 03 07:51:01.409448 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.409415    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-openshift-ca" pod "d4abaca6-d153-428d-8f2a-f2c1628046b9" (UID: "d4abaca6-d153-428d-8f2a-f2c1628046b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:51:01.411441 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.411417    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4abaca6-d153-428d-8f2a-f2c1628046b9-kube-api-access-lrkbr" pod "d4abaca6-d153-428d-8f2a-f2c1628046b9" (UID: "d4abaca6-d153-428d-8f2a-f2c1628046b9"). InnerVolumeSpecName "kube-api-access-lrkbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:51:01.411543 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.411452    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-xks-ca" pod "d4abaca6-d153-428d-8f2a-f2c1628046b9" (UID: "d4abaca6-d153-428d-8f2a-f2c1628046b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:51:01.510110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.510072    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrkbr\" (UniqueName: \"kubernetes.io/projected/d4abaca6-d153-428d-8f2a-f2c1628046b9-kube-api-access-lrkbr\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:51:01.510110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.510103    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:51:01.510110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.510113    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4abaca6-d153-428d-8f2a-f2c1628046b9-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:51:01.805488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.805409    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28386bf2-f823-49d3-bf34-06145054f1bc" path="/var/lib/kubelet/pods/28386bf2-f823-49d3-bf34-06145054f1bc/volumes"
Nov 03 07:51:01.805831 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:01.805819    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a34be0f8-6024-477b-864d-6eff1225d453" path="/var/lib/kubelet/pods/a34be0f8-6024-477b-864d-6eff1225d453/volumes"
Nov 03 07:51:02.153916 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:02.153879    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92" event={"ID":"d4abaca6-d153-428d-8f2a-f2c1628046b9","Type":"ContainerDied","Data":"26c5eb3d379810d9fbe365948878ba2c0e9264e00e8c445bc51368a07aaa30b1"}
Nov 03 07:51:02.154102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:02.153939    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26c5eb3d379810d9fbe365948878ba2c0e9264e00e8c445bc51368a07aaa30b1"
Nov 03 07:51:02.154102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:02.153892    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"
Nov 03 07:51:04.759912 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:51:04.759874    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:51:29.064727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.064696    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"]
Nov 03 07:51:29.067408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.065150    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d4abaca6-d153-428d-8f2a-f2c1628046b9" containerName="revoke-keys"
Nov 03 07:51:29.067408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.065161    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4abaca6-d153-428d-8f2a-f2c1628046b9" containerName="revoke-keys"
Nov 03 07:51:29.067408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.065231    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4abaca6-d153-428d-8f2a-f2c1628046b9" containerName="revoke-keys"
Nov 03 07:51:29.068489 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.068466    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:29.108930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.108888    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpb44\" (UniqueName: \"kubernetes.io/projected/4ad1392a-cf6e-4aa0-b451-6355a498b071-kube-api-access-wpb44\") pod \"maas-api-revoke-sub-2e1f5b5f-djjnm\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:29.108930 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.108939    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-djjnm\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:29.109256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.109020    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-djjnm\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:29.127967 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.127921    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"]
Nov 03 07:51:29.174780 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.174720    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:51:29.174780 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.174768    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:51:29.174780 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.174745    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:51:29.175102 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.174880    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:51:29.210256 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.210218    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpb44\" (UniqueName: \"kubernetes.io/projected/4ad1392a-cf6e-4aa0-b451-6355a498b071-kube-api-access-wpb44\") pod \"maas-api-revoke-sub-2e1f5b5f-djjnm\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:29.210432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.210277    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-djjnm\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:29.210432 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.210313    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-djjnm\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:29.210979 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.210956    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-djjnm\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:29.212768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.212749    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-djjnm\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:29.275016 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.274988    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpb44\" (UniqueName: \"kubernetes.io/projected/4ad1392a-cf6e-4aa0-b451-6355a498b071-kube-api-access-wpb44\") pod \"maas-api-revoke-sub-2e1f5b5f-djjnm\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:29.381333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.381256    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:29.756641 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:29.756616    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"]
Nov 03 07:51:29.757954 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:51:29.757927    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad1392a_cf6e_4aa0_b451_6355a498b071.slice/crio-a880047e7f3600d0a9ab31b99ec29189145bf34add48d7470878d51a74546983 WatchSource:0}: Error finding container a880047e7f3600d0a9ab31b99ec29189145bf34add48d7470878d51a74546983: Status 404 returned error can't find the container with id a880047e7f3600d0a9ab31b99ec29189145bf34add48d7470878d51a74546983
Nov 03 07:51:30.055722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:30.055695    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"]
Nov 03 07:51:30.058399 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:30.058375    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"]
Nov 03 07:51:30.061021 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:30.061000    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af13e332-2zxmv"]
Nov 03 07:51:30.106106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:30.106080    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fba7b73-zr6v7"]
Nov 03 07:51:30.330052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:30.330012    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm" event={"ID":"4ad1392a-cf6e-4aa0-b451-6355a498b071","Type":"ContainerStarted","Data":"9582816c391f5c5635b39c0d00d274089337b8229c656e4d74a604da4ad17153"}
Nov 03 07:51:30.330285 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:30.330070    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm" event={"ID":"4ad1392a-cf6e-4aa0-b451-6355a498b071","Type":"ContainerStarted","Data":"a880047e7f3600d0a9ab31b99ec29189145bf34add48d7470878d51a74546983"}
Nov 03 07:51:30.451350 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:30.451296    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm" podStartSLOduration=1.451280715 podStartE2EDuration="1.451280715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:51:30.450859001 +0000 UTC m=+3142.783227997" watchObservedRunningTime="2026-11-03 07:51:30.451280715 +0000 UTC m=+3142.783649709"
Nov 03 07:51:31.336816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:31.336777    4470 generic.go:356] "Generic (PLEG): container finished" podID="4ad1392a-cf6e-4aa0-b451-6355a498b071" containerID="9582816c391f5c5635b39c0d00d274089337b8229c656e4d74a604da4ad17153" exitCode=0
Nov 03 07:51:31.337236 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:31.336821    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm" event={"ID":"4ad1392a-cf6e-4aa0-b451-6355a498b071","Type":"ContainerDied","Data":"9582816c391f5c5635b39c0d00d274089337b8229c656e4d74a604da4ad17153"}
Nov 03 07:51:31.804893 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:31.804819    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745e4642-5846-4cbb-bafc-9c2deb9f51bf" path="/var/lib/kubelet/pods/745e4642-5846-4cbb-bafc-9c2deb9f51bf/volumes"
Nov 03 07:51:31.805248 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:31.805234    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db17c026-3654-4351-b8d1-8f0d505626bd" path="/var/lib/kubelet/pods/db17c026-3654-4351-b8d1-8f0d505626bd/volumes"
Nov 03 07:51:32.488003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:32.487980    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:32.543067 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:32.543036    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad1392a-cf6e-4aa0-b451-6355a498b071-kube-api-access-wpb44\" (UniqueName: \"kubernetes.io/projected/4ad1392a-cf6e-4aa0-b451-6355a498b071-kube-api-access-wpb44\") pod \"4ad1392a-cf6e-4aa0-b451-6355a498b071\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") "
Nov 03 07:51:32.543251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:32.543148    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-xks-ca\") pod \"4ad1392a-cf6e-4aa0-b451-6355a498b071\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") "
Nov 03 07:51:32.543251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:32.543197    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-openshift-ca\") pod \"4ad1392a-cf6e-4aa0-b451-6355a498b071\" (UID: \"4ad1392a-cf6e-4aa0-b451-6355a498b071\") "
Nov 03 07:51:32.543609 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:32.543567    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-openshift-ca" pod "4ad1392a-cf6e-4aa0-b451-6355a498b071" (UID: "4ad1392a-cf6e-4aa0-b451-6355a498b071"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:51:32.545191 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:32.545167    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad1392a-cf6e-4aa0-b451-6355a498b071-kube-api-access-wpb44" pod "4ad1392a-cf6e-4aa0-b451-6355a498b071" (UID: "4ad1392a-cf6e-4aa0-b451-6355a498b071"). InnerVolumeSpecName "kube-api-access-wpb44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:51:32.545330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:32.545308    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-xks-ca" pod "4ad1392a-cf6e-4aa0-b451-6355a498b071" (UID: "4ad1392a-cf6e-4aa0-b451-6355a498b071"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:51:32.645166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:32.645074    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:51:32.645166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:32.645104    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad1392a-cf6e-4aa0-b451-6355a498b071-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:51:32.645166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:32.645114    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpb44\" (UniqueName: \"kubernetes.io/projected/4ad1392a-cf6e-4aa0-b451-6355a498b071-kube-api-access-wpb44\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:51:33.351276 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:33.351242    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm" event={"ID":"4ad1392a-cf6e-4aa0-b451-6355a498b071","Type":"ContainerDied","Data":"a880047e7f3600d0a9ab31b99ec29189145bf34add48d7470878d51a74546983"}
Nov 03 07:51:33.351460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:33.351291    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a880047e7f3600d0a9ab31b99ec29189145bf34add48d7470878d51a74546983"
Nov 03 07:51:33.351460 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:33.351252    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"
Nov 03 07:51:34.759695 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:51:34.759660    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:51:41.603279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.603235    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"]
Nov 03 07:51:41.603898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.603877    4470 cpu_manager.go:414] "containerMap: removing container" podUID="4ad1392a-cf6e-4aa0-b451-6355a498b071" containerName="revoke-keys"
Nov 03 07:51:41.603956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.603903    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad1392a-cf6e-4aa0-b451-6355a498b071" containerName="revoke-keys"
Nov 03 07:51:41.604064 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.604053    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad1392a-cf6e-4aa0-b451-6355a498b071" containerName="revoke-keys"
Nov 03 07:51:41.607628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.607603    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:41.665380 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.665344    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"]
Nov 03 07:51:41.710172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.710124    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:51:41.710172 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.710126    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:51:41.710404 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.710220    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:51:41.712426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.712404    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:51:41.736204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.736162    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ggd\" (UniqueName: \"kubernetes.io/projected/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-kube-api-access-v8ggd\") pod \"maas-api-revoke-sub-ec843018-jj6gg\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:41.736368 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.736230    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jj6gg\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:41.736417 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.736401    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jj6gg\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:41.837749 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.837719    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jj6gg\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:41.837953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.837765    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ggd\" (UniqueName: \"kubernetes.io/projected/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-kube-api-access-v8ggd\") pod \"maas-api-revoke-sub-ec843018-jj6gg\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:41.837953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.837796    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jj6gg\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:41.838535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.838510    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jj6gg\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:41.840229 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.840210    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jj6gg\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:41.902935 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.902865    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ggd\" (UniqueName: \"kubernetes.io/projected/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-kube-api-access-v8ggd\") pod \"maas-api-revoke-sub-ec843018-jj6gg\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:41.920745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:41.920713    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:42.304713 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:42.304682    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"]
Nov 03 07:51:42.305040 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:51:42.305004    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9f0349_0a60_4cd4_9bcd_3f595d8b574e.slice/crio-199b14497fc442ec7a91995c0e4a1acb9d827aef3c3bb096f71d8645b1ad1a04 WatchSource:0}: Error finding container 199b14497fc442ec7a91995c0e4a1acb9d827aef3c3bb096f71d8645b1ad1a04: Status 404 returned error can't find the container with id 199b14497fc442ec7a91995c0e4a1acb9d827aef3c3bb096f71d8645b1ad1a04
Nov 03 07:51:42.403199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:42.403165    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg" event={"ID":"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e","Type":"ContainerStarted","Data":"199b14497fc442ec7a91995c0e4a1acb9d827aef3c3bb096f71d8645b1ad1a04"}
Nov 03 07:51:43.410942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:43.410863    4470 generic.go:356] "Generic (PLEG): container finished" podID="0f9f0349-0a60-4cd4-9bcd-3f595d8b574e" containerID="49f61781ecf78b16cc0a66614343865fbad14190db568239dbfc839c894c6129" exitCode=0
Nov 03 07:51:43.410942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:43.410924    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg" event={"ID":"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e","Type":"ContainerDied","Data":"49f61781ecf78b16cc0a66614343865fbad14190db568239dbfc839c894c6129"}
Nov 03 07:51:44.550174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:44.550150    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:44.666119 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:44.666085    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-xks-ca\") pod \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") "
Nov 03 07:51:44.666329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:44.666217    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-openshift-ca\") pod \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") "
Nov 03 07:51:44.666329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:44.666266    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-kube-api-access-v8ggd\" (UniqueName: \"kubernetes.io/projected/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-kube-api-access-v8ggd\") pod \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\" (UID: \"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e\") "
Nov 03 07:51:44.666664 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:44.666602    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-openshift-ca" pod "0f9f0349-0a60-4cd4-9bcd-3f595d8b574e" (UID: "0f9f0349-0a60-4cd4-9bcd-3f595d8b574e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:51:44.668555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:44.668529    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-kube-api-access-v8ggd" pod "0f9f0349-0a60-4cd4-9bcd-3f595d8b574e" (UID: "0f9f0349-0a60-4cd4-9bcd-3f595d8b574e"). InnerVolumeSpecName "kube-api-access-v8ggd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:51:44.668555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:44.668532    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-xks-ca" pod "0f9f0349-0a60-4cd4-9bcd-3f595d8b574e" (UID: "0f9f0349-0a60-4cd4-9bcd-3f595d8b574e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:51:44.768118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:44.768084    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:51:44.768118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:44.768114    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:51:44.768118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:44.768124    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ggd\" (UniqueName: \"kubernetes.io/projected/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e-kube-api-access-v8ggd\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:51:45.423290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:45.423261    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"
Nov 03 07:51:45.423290 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:45.423284    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg" event={"ID":"0f9f0349-0a60-4cd4-9bcd-3f595d8b574e","Type":"ContainerDied","Data":"199b14497fc442ec7a91995c0e4a1acb9d827aef3c3bb096f71d8645b1ad1a04"}
Nov 03 07:51:45.423570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:45.423333    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="199b14497fc442ec7a91995c0e4a1acb9d827aef3c3bb096f71d8645b1ad1a04"
Nov 03 07:51:50.054824 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:50.054777    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"]
Nov 03 07:51:50.105457 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:50.105426    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebbcd033-x5jtg"]
Nov 03 07:51:51.805009 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:51:51.804964    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f44b1f-4fc6-4f14-9e34-6f1230d5076f" path="/var/lib/kubelet/pods/19f44b1f-4fc6-4f14-9e34-6f1230d5076f/volumes"
Nov 03 07:52:04.759123 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:52:04.759075    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:52:14.932008 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:14.931975    4470 scope.go:122] "RemoveContainer" containerID="d6ac8b94ceb3866c529535b7c4b21c4abf57485e26c2c658632605f1ce2edcac"
Nov 03 07:52:14.941013 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:14.940991    4470 scope.go:122] "RemoveContainer" containerID="2a6b5d34ea5111c031e22dc4269d3f9e349db00f58dd4e409874cf1dc2919a32"
Nov 03 07:52:14.949759 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:14.949736    4470 scope.go:122] "RemoveContainer" containerID="47133ac6e334690d544058d2fa2c626ab42796667ea51999c69294aa77cb9717"
Nov 03 07:52:14.958196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:14.958171    4470 scope.go:122] "RemoveContainer" containerID="d948b1489588309c554ae901aa550dd09a4a08b16d5946c808afb174a9357614"
Nov 03 07:52:14.966257 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:14.966235    4470 scope.go:122] "RemoveContainer" containerID="ee79d59b6f9152bce69f7add9e95aa3b16edb9f67f3d9084d5d0e94362a549a0"
Nov 03 07:52:14.974206 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:14.974186    4470 scope.go:122] "RemoveContainer" containerID="9c1e246a28bc18fe4738090959aac847104136ed03eb3245432377ede3ff2d2b"
Nov 03 07:52:14.982451 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:14.982430    4470 scope.go:122] "RemoveContainer" containerID="03b975d2c518f039c4dce91650083cd11f0b3019ff35a05b8fb45a3a2b1aecd5"
Nov 03 07:52:17.338065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.334768    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"]
Nov 03 07:52:17.338065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.335758    4470 cpu_manager.go:414] "containerMap: removing container" podUID="0f9f0349-0a60-4cd4-9bcd-3f595d8b574e" containerName="revoke-keys"
Nov 03 07:52:17.338065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.335780    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9f0349-0a60-4cd4-9bcd-3f595d8b574e" containerName="revoke-keys"
Nov 03 07:52:17.338065 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.336018    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9f0349-0a60-4cd4-9bcd-3f595d8b574e" containerName="revoke-keys"
Nov 03 07:52:17.342401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.342374    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:17.395661 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.395631    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"]
Nov 03 07:52:17.443909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.443878    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:52:17.443909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.443890    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:52:17.443909 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.443890    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:52:17.447917 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.447902    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:52:17.459958 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.459932    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-bzn5w\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:17.460071 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.460057    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmd89\" (UniqueName: \"kubernetes.io/projected/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-kube-api-access-fmd89\") pod \"maas-api-revoke-sub-046e1343-bzn5w\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:17.460120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.460109    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-bzn5w\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:17.560911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.560869    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmd89\" (UniqueName: \"kubernetes.io/projected/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-kube-api-access-fmd89\") pod \"maas-api-revoke-sub-046e1343-bzn5w\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:17.560911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.560907    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-bzn5w\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:17.561190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.560986    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-bzn5w\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:17.561679 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.561658    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-bzn5w\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:17.563449 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.563424    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-bzn5w\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:17.625695 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.625612    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmd89\" (UniqueName: \"kubernetes.io/projected/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-kube-api-access-fmd89\") pod \"maas-api-revoke-sub-046e1343-bzn5w\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:17.658083 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:17.658051    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:18.037863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:18.037830    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"]
Nov 03 07:52:18.038998 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:52:18.038972    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66d55a6_c31d_4a2a_952d_3873e5f7e2ad.slice/crio-82634ed0e075421ef6131e1675efe01e5ac4c37dab7cf24af312871ee9ee8365 WatchSource:0}: Error finding container 82634ed0e075421ef6131e1675efe01e5ac4c37dab7cf24af312871ee9ee8365: Status 404 returned error can't find the container with id 82634ed0e075421ef6131e1675efe01e5ac4c37dab7cf24af312871ee9ee8365
Nov 03 07:52:18.651052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:18.650936    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w" event={"ID":"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad","Type":"ContainerStarted","Data":"ec1f1d6f7f772bc19e02427f212f3c499a7031c908fe96a71fcf6464fa038a41"}
Nov 03 07:52:18.651052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:18.650995    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w" event={"ID":"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad","Type":"ContainerStarted","Data":"82634ed0e075421ef6131e1675efe01e5ac4c37dab7cf24af312871ee9ee8365"}
Nov 03 07:52:18.770906 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:18.770853    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w" podStartSLOduration=1.770837909 podStartE2EDuration="1.770837909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:52:18.769988663 +0000 UTC m=+3191.102357656" watchObservedRunningTime="2026-11-03 07:52:18.770837909 +0000 UTC m=+3191.103206902"
Nov 03 07:52:19.658107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:19.658072    4470 generic.go:356] "Generic (PLEG): container finished" podID="d66d55a6-c31d-4a2a-952d-3873e5f7e2ad" containerID="ec1f1d6f7f772bc19e02427f212f3c499a7031c908fe96a71fcf6464fa038a41" exitCode=0
Nov 03 07:52:19.658584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:19.658161    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w" event={"ID":"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad","Type":"ContainerDied","Data":"ec1f1d6f7f772bc19e02427f212f3c499a7031c908fe96a71fcf6464fa038a41"}
Nov 03 07:52:20.801208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:20.801181    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:20.896619 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:20.896590    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-xks-ca\") pod \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") "
Nov 03 07:52:20.896832 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:20.896674    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-openshift-ca\") pod \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") "
Nov 03 07:52:20.896832 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:20.896706    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-kube-api-access-fmd89\" (UniqueName: \"kubernetes.io/projected/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-kube-api-access-fmd89\") pod \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\" (UID: \"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad\") "
Nov 03 07:52:20.897038 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:20.897013    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-openshift-ca" pod "d66d55a6-c31d-4a2a-952d-3873e5f7e2ad" (UID: "d66d55a6-c31d-4a2a-952d-3873e5f7e2ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:52:20.897204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:20.897186    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:52:20.898999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:20.898976    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-xks-ca" pod "d66d55a6-c31d-4a2a-952d-3873e5f7e2ad" (UID: "d66d55a6-c31d-4a2a-952d-3873e5f7e2ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:52:20.899148 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:20.899117    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-kube-api-access-fmd89" pod "d66d55a6-c31d-4a2a-952d-3873e5f7e2ad" (UID: "d66d55a6-c31d-4a2a-952d-3873e5f7e2ad"). InnerVolumeSpecName "kube-api-access-fmd89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:52:20.997842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:20.997753    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmd89\" (UniqueName: \"kubernetes.io/projected/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-kube-api-access-fmd89\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:52:20.997842 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:20.997787    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:52:21.670371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:21.670339    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"
Nov 03 07:52:21.670578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:21.670370    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w" event={"ID":"d66d55a6-c31d-4a2a-952d-3873e5f7e2ad","Type":"ContainerDied","Data":"82634ed0e075421ef6131e1675efe01e5ac4c37dab7cf24af312871ee9ee8365"}
Nov 03 07:52:21.670578 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:21.670413    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82634ed0e075421ef6131e1675efe01e5ac4c37dab7cf24af312871ee9ee8365"
Nov 03 07:52:28.115269 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.115230    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"]
Nov 03 07:52:28.115685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.115671    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d66d55a6-c31d-4a2a-952d-3873e5f7e2ad" containerName="revoke-keys"
Nov 03 07:52:28.115732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.115686    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d66d55a6-c31d-4a2a-952d-3873e5f7e2ad" containerName="revoke-keys"
Nov 03 07:52:28.115769 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.115748    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d66d55a6-c31d-4a2a-952d-3873e5f7e2ad" containerName="revoke-keys"
Nov 03 07:52:28.119109 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.119087    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:28.172199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.172161    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-wwpkk\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:28.172389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.172213    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96kdd\" (UniqueName: \"kubernetes.io/projected/91bb7baf-fbe2-4106-9296-7294cdaa1e43-kube-api-access-96kdd\") pod \"maas-api-revoke-sub-d3fba056-wwpkk\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:28.172389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.172339    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-wwpkk\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:28.179003 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.178963    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"]
Nov 03 07:52:28.225944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.225914    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:52:28.226129 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.226108    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:52:28.226205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.226108    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:52:28.226205 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.226173    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:52:28.273507 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.273474    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-wwpkk\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:28.273703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.273560    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-wwpkk\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:28.273703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.273603    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96kdd\" (UniqueName: \"kubernetes.io/projected/91bb7baf-fbe2-4106-9296-7294cdaa1e43-kube-api-access-96kdd\") pod \"maas-api-revoke-sub-d3fba056-wwpkk\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:28.274207 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.274182    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-wwpkk\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:28.275890 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.275872    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-wwpkk\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:28.337887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.337853    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96kdd\" (UniqueName: \"kubernetes.io/projected/91bb7baf-fbe2-4106-9296-7294cdaa1e43-kube-api-access-96kdd\") pod \"maas-api-revoke-sub-d3fba056-wwpkk\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:28.432330 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.432244    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:28.813391 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:28.813365    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"]
Nov 03 07:52:28.815007 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:52:28.814975    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91bb7baf_fbe2_4106_9296_7294cdaa1e43.slice/crio-0ff89fe0fb08a1bb421700a714f27ecf69cf94872ea3ee30a48518ad093ec91a WatchSource:0}: Error finding container 0ff89fe0fb08a1bb421700a714f27ecf69cf94872ea3ee30a48518ad093ec91a: Status 404 returned error can't find the container with id 0ff89fe0fb08a1bb421700a714f27ecf69cf94872ea3ee30a48518ad093ec91a
Nov 03 07:52:29.712220 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:29.712183    4470 generic.go:356] "Generic (PLEG): container finished" podID="91bb7baf-fbe2-4106-9296-7294cdaa1e43" containerID="df4295737be585f244f8f3da6a78bac1dabdd20ecef9eb78060998f5b251ce7e" exitCode=0
Nov 03 07:52:29.712610 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:29.712275    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk" event={"ID":"91bb7baf-fbe2-4106-9296-7294cdaa1e43","Type":"ContainerDied","Data":"df4295737be585f244f8f3da6a78bac1dabdd20ecef9eb78060998f5b251ce7e"}
Nov 03 07:52:29.712610 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:29.712327    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk" event={"ID":"91bb7baf-fbe2-4106-9296-7294cdaa1e43","Type":"ContainerStarted","Data":"0ff89fe0fb08a1bb421700a714f27ecf69cf94872ea3ee30a48518ad093ec91a"}
Nov 03 07:52:30.923051 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:30.923016    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:31.000487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.000376    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-openshift-ca\") pod \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") "
Nov 03 07:52:31.000487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.000475    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91bb7baf-fbe2-4106-9296-7294cdaa1e43-kube-api-access-96kdd\" (UniqueName: \"kubernetes.io/projected/91bb7baf-fbe2-4106-9296-7294cdaa1e43-kube-api-access-96kdd\") pod \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") "
Nov 03 07:52:31.000757 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.000592    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-xks-ca\") pod \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\" (UID: \"91bb7baf-fbe2-4106-9296-7294cdaa1e43\") "
Nov 03 07:52:31.001001 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.000839    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-openshift-ca" pod "91bb7baf-fbe2-4106-9296-7294cdaa1e43" (UID: "91bb7baf-fbe2-4106-9296-7294cdaa1e43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:52:31.001001 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.000970    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:52:31.003788 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.003756    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91bb7baf-fbe2-4106-9296-7294cdaa1e43-kube-api-access-96kdd" pod "91bb7baf-fbe2-4106-9296-7294cdaa1e43" (UID: "91bb7baf-fbe2-4106-9296-7294cdaa1e43"). InnerVolumeSpecName "kube-api-access-96kdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:52:31.004260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.004233    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-xks-ca" pod "91bb7baf-fbe2-4106-9296-7294cdaa1e43" (UID: "91bb7baf-fbe2-4106-9296-7294cdaa1e43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:52:31.102185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.102124    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96kdd\" (UniqueName: \"kubernetes.io/projected/91bb7baf-fbe2-4106-9296-7294cdaa1e43-kube-api-access-96kdd\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:52:31.102185 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.102185    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91bb7baf-fbe2-4106-9296-7294cdaa1e43-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:52:31.751169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.745609    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk" event={"ID":"91bb7baf-fbe2-4106-9296-7294cdaa1e43","Type":"ContainerDied","Data":"0ff89fe0fb08a1bb421700a714f27ecf69cf94872ea3ee30a48518ad093ec91a"}
Nov 03 07:52:31.751169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.745675    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ff89fe0fb08a1bb421700a714f27ecf69cf94872ea3ee30a48518ad093ec91a"
Nov 03 07:52:31.751169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:31.745747    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"
Nov 03 07:52:32.081902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:32.081182    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"]
Nov 03 07:52:32.139333 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:32.139293    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1798466-gsfwz"]
Nov 03 07:52:33.804733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:52:33.804698    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1474753-9572-4e40-86bf-dc84143e62fa" path="/var/lib/kubelet/pods/f1474753-9572-4e40-86bf-dc84143e62fa/volumes"
Nov 03 07:52:34.758899 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:52:34.758867    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:53:02.952867 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:02.952827    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"]
Nov 03 07:53:02.953458 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:02.953441    4470 cpu_manager.go:414] "containerMap: removing container" podUID="91bb7baf-fbe2-4106-9296-7294cdaa1e43" containerName="revoke-keys"
Nov 03 07:53:02.953524 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:02.953460    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91bb7baf-fbe2-4106-9296-7294cdaa1e43" containerName="revoke-keys"
Nov 03 07:53:02.953566 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:02.953537    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="91bb7baf-fbe2-4106-9296-7294cdaa1e43" containerName="revoke-keys"
Nov 03 07:53:02.956993 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:02.956969    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:03.016572 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.016541    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"]
Nov 03 07:53:03.051052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.051014    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw22n\" (UniqueName: \"kubernetes.io/projected/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-kube-api-access-qw22n\") pod \"maas-api-revoke-sub-ae594979-nbpz9\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:03.051251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.051175    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nbpz9\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:03.051251 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.051223    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nbpz9\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:03.063554 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.063519    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:53:03.063554 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.063524    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:53:03.063554 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.063527    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:53:03.063789 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.063578    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:53:03.152819 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.152780    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nbpz9\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:03.153002 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.152838    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nbpz9\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:03.153002 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.152923    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw22n\" (UniqueName: \"kubernetes.io/projected/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-kube-api-access-qw22n\") pod \"maas-api-revoke-sub-ae594979-nbpz9\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:03.153508 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.153487    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nbpz9\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:03.155455 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.155435    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nbpz9\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:03.217527 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.217443    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw22n\" (UniqueName: \"kubernetes.io/projected/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-kube-api-access-qw22n\") pod \"maas-api-revoke-sub-ae594979-nbpz9\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:03.270041 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.270001    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:03.646691 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.646663    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"]
Nov 03 07:53:03.648097 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:53:03.648074    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e0ef2f_0226_4f9a_98a5_f4c49a43b80a.slice/crio-d15d1aef878f8751578feabf7796fff510fc9c17b6330e8bfab38ebd757b9359 WatchSource:0}: Error finding container d15d1aef878f8751578feabf7796fff510fc9c17b6330e8bfab38ebd757b9359: Status 404 returned error can't find the container with id d15d1aef878f8751578feabf7796fff510fc9c17b6330e8bfab38ebd757b9359
Nov 03 07:53:03.975654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.975618    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9" event={"ID":"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a","Type":"ContainerStarted","Data":"4a301be39b0d514a460ca045dbad50e6ccb91b8f5051d25d1a67f56e2c09d512"}
Nov 03 07:53:03.976005 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:03.975699    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9" event={"ID":"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a","Type":"ContainerStarted","Data":"d15d1aef878f8751578feabf7796fff510fc9c17b6330e8bfab38ebd757b9359"}
Nov 03 07:53:04.054550 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:04.054518    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"]
Nov 03 07:53:04.062721 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:04.062689    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"]
Nov 03 07:53:04.082514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:04.082487    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-647c1c93-6lgw7"]
Nov 03 07:53:04.084361 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:04.084343    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eded570c-l7b7v"]
Nov 03 07:53:04.103117 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:04.103075    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9" podStartSLOduration=2.103063491 podStartE2EDuration="2.103063491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:04.102877587 +0000 UTC m=+3236.435246581" watchObservedRunningTime="2026-11-03 07:53:04.103063491 +0000 UTC m=+3236.435432485"
Nov 03 07:53:04.758676 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:53:04.758644    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:53:04.982590 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:04.982552    4470 generic.go:356] "Generic (PLEG): container finished" podID="a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a" containerID="4a301be39b0d514a460ca045dbad50e6ccb91b8f5051d25d1a67f56e2c09d512" exitCode=0
Nov 03 07:53:04.983104 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:04.982641    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9" event={"ID":"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a","Type":"ContainerDied","Data":"4a301be39b0d514a460ca045dbad50e6ccb91b8f5051d25d1a67f56e2c09d512"}
Nov 03 07:53:05.805212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:05.805184    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="602e922a-3f3e-4388-a0f5-00d70b163d24" path="/var/lib/kubelet/pods/602e922a-3f3e-4388-a0f5-00d70b163d24/volumes"
Nov 03 07:53:05.805581 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:05.805568    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c670d8f1-5cc5-4519-b051-c7faeaeabb37" path="/var/lib/kubelet/pods/c670d8f1-5cc5-4519-b051-c7faeaeabb37/volumes"
Nov 03 07:53:06.126090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.126060    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:06.285196 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.285166    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-openshift-ca\") pod \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") "
Nov 03 07:53:06.285396 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.285229    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-kube-api-access-qw22n\" (UniqueName: \"kubernetes.io/projected/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-kube-api-access-qw22n\") pod \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") "
Nov 03 07:53:06.285396 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.285307    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-xks-ca\") pod \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\" (UID: \"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a\") "
Nov 03 07:53:06.285545 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.285521    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-openshift-ca" pod "a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a" (UID: "a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:53:06.285683 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.285668    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:53:06.287668 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.287643    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-xks-ca" pod "a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a" (UID: "a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:53:06.287814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.287697    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-kube-api-access-qw22n" pod "a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a" (UID: "a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a"). InnerVolumeSpecName "kube-api-access-qw22n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:53:06.386629 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.386566    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw22n\" (UniqueName: \"kubernetes.io/projected/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-kube-api-access-qw22n\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:53:06.386629 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.386591    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:53:06.994911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.994879    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"
Nov 03 07:53:06.994911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.994903    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9" event={"ID":"a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a","Type":"ContainerDied","Data":"d15d1aef878f8751578feabf7796fff510fc9c17b6330e8bfab38ebd757b9359"}
Nov 03 07:53:06.995210 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:06.994950    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d15d1aef878f8751578feabf7796fff510fc9c17b6330e8bfab38ebd757b9359"
Nov 03 07:53:15.031779 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:15.031747    4470 scope.go:122] "RemoveContainer" containerID="d5ea61cb6c7d5198c89fb188ab1f4a4bc7857e4c7961c00ec3ee7b59e140e170"
Nov 03 07:53:15.041613 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:15.041591    4470 scope.go:122] "RemoveContainer" containerID="4e14e5711f185cde54ae64e9f6a93c2e638606e952f000d47666f6375905200e"
Nov 03 07:53:15.050107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:15.050091    4470 scope.go:122] "RemoveContainer" containerID="c223c845d3ad4b664e465c367dd0448d773d26ba1b3b7ac16608c3d825e0a681"
Nov 03 07:53:15.057964 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:15.057941    4470 scope.go:122] "RemoveContainer" containerID="e0eab36e389a0d46f5bc0ce10ba34300b7794eee2cb27e1fa00814b2b25ca79b"
Nov 03 07:53:31.107216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.107173    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"]
Nov 03 07:53:31.107870 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.107848    4470 cpu_manager.go:414] "containerMap: removing container" podUID="a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a" containerName="revoke-keys"
Nov 03 07:53:31.107951 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.107875    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a" containerName="revoke-keys"
Nov 03 07:53:31.107987 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.107979    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a" containerName="revoke-keys"
Nov 03 07:53:31.112695 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.112675    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:31.170255 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.170225    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"]
Nov 03 07:53:31.192948 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.192912    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvcd8\" (UniqueName: \"kubernetes.io/projected/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-kube-api-access-vvcd8\") pod \"maas-api-revoke-sub-a4aef7d7-slz5k\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:31.193159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.192953    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-slz5k\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:31.193282 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.193262    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-slz5k\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:31.217663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.217634    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:53:31.217663 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.217650    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:53:31.217865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.217704    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:53:31.217865 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.217783    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:53:31.293917 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.293873    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-slz5k\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:31.294083 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.293928    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcd8\" (UniqueName: \"kubernetes.io/projected/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-kube-api-access-vvcd8\") pod \"maas-api-revoke-sub-a4aef7d7-slz5k\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:31.294083 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.293962    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-slz5k\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:31.294714 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.294691    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-slz5k\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:31.296370 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.296352    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-slz5k\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:31.357963 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.357897    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcd8\" (UniqueName: \"kubernetes.io/projected/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-kube-api-access-vvcd8\") pod \"maas-api-revoke-sub-a4aef7d7-slz5k\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:31.430633 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.430600    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:31.809588 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:31.809550    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"]
Nov 03 07:53:31.810699 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:53:31.810674    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4db0cf_62c7_4c0e_b1b6_9db15eb033e0.slice/crio-074d0beeabb5441ee0ac1fc3a437fb1cbc8183af86c0d15c1295b02e3996dd5d WatchSource:0}: Error finding container 074d0beeabb5441ee0ac1fc3a437fb1cbc8183af86c0d15c1295b02e3996dd5d: Status 404 returned error can't find the container with id 074d0beeabb5441ee0ac1fc3a437fb1cbc8183af86c0d15c1295b02e3996dd5d
Nov 03 07:53:32.174426 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:32.174390    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k" event={"ID":"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0","Type":"ContainerStarted","Data":"5cf81e68862e8da02cf33669cb075375d77e44e4bc7695454e0da46588e482c6"}
Nov 03 07:53:32.174854 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:32.174446    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k" event={"ID":"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0","Type":"ContainerStarted","Data":"074d0beeabb5441ee0ac1fc3a437fb1cbc8183af86c0d15c1295b02e3996dd5d"}
Nov 03 07:53:32.294882 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:32.294828    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k" podStartSLOduration=1.294812743 podStartE2EDuration="1.294812743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:32.29416655 +0000 UTC m=+3264.626535545" watchObservedRunningTime="2026-11-03 07:53:32.294812743 +0000 UTC m=+3264.627181736"
Nov 03 07:53:33.180992 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:33.180957    4470 generic.go:356] "Generic (PLEG): container finished" podID="9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0" containerID="5cf81e68862e8da02cf33669cb075375d77e44e4bc7695454e0da46588e482c6" exitCode=0
Nov 03 07:53:33.181444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:33.181049    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k" event={"ID":"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0","Type":"ContainerDied","Data":"5cf81e68862e8da02cf33669cb075375d77e44e4bc7695454e0da46588e482c6"}
Nov 03 07:53:34.325577 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:34.325547    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:34.425840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:34.425805    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-kube-api-access-vvcd8\" (UniqueName: \"kubernetes.io/projected/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-kube-api-access-vvcd8\") pod \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") "
Nov 03 07:53:34.426011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:34.425875    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-xks-ca\") pod \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") "
Nov 03 07:53:34.426011 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:34.425903    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-openshift-ca\") pod \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\" (UID: \"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0\") "
Nov 03 07:53:34.426402 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:34.426375    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-openshift-ca" pod "9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0" (UID: "9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:53:34.428202 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:34.428172    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-xks-ca" pod "9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0" (UID: "9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:53:34.428304 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:34.428269    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-kube-api-access-vvcd8" pod "9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0" (UID: "9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0"). InnerVolumeSpecName "kube-api-access-vvcd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:53:34.527407 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:34.527323    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:53:34.527407 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:34.527350    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:53:34.527407 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:34.527360    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvcd8\" (UniqueName: \"kubernetes.io/projected/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0-kube-api-access-vvcd8\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:53:34.759624 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:53:34.759573    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:53:35.199659 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:35.199618    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k" event={"ID":"9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0","Type":"ContainerDied","Data":"074d0beeabb5441ee0ac1fc3a437fb1cbc8183af86c0d15c1295b02e3996dd5d"}
Nov 03 07:53:35.199847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:35.199665    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"
Nov 03 07:53:35.199847 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:53:35.199694    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="074d0beeabb5441ee0ac1fc3a437fb1cbc8183af86c0d15c1295b02e3996dd5d"
Nov 03 07:54:00.060697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:00.060604    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"]
Nov 03 07:54:00.111402 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:00.111370    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0798fa-bpw6j"]
Nov 03 07:54:01.775468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.775389    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"]
Nov 03 07:54:01.775891 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.775876    4470 cpu_manager.go:414] "containerMap: removing container" podUID="9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0" containerName="revoke-keys"
Nov 03 07:54:01.775937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.775893    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0" containerName="revoke-keys"
Nov 03 07:54:01.775985 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.775970    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0" containerName="revoke-keys"
Nov 03 07:54:01.779215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.779193    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:01.804654 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.804621    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e765df-1366-4bf0-8b4f-38bed6cfe159" path="/var/lib/kubelet/pods/e3e765df-1366-4bf0-8b4f-38bed6cfe159/volumes"
Nov 03 07:54:01.834779 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.834753    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:54:01.850414 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.850378    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-d9v4z\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:01.850601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.850429    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxtnp\" (UniqueName: \"kubernetes.io/projected/994e457f-8114-4391-b93c-3ce35edc5543-kube-api-access-wxtnp\") pod \"maas-api-revoke-sub-79805ca9-d9v4z\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:01.850601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.850591    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-d9v4z\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:01.884279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.884248    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:54:01.884439 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.884352    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:54:01.884486 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.884456    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:54:01.886971 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.886950    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"]
Nov 03 07:54:01.952061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.952026    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-d9v4z\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:01.952061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.952068    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxtnp\" (UniqueName: \"kubernetes.io/projected/994e457f-8114-4391-b93c-3ce35edc5543-kube-api-access-wxtnp\") pod \"maas-api-revoke-sub-79805ca9-d9v4z\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:01.952300 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.952123    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-d9v4z\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:01.952777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.952749    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-d9v4z\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:01.954631 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:01.954610    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-d9v4z\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:02.016488 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:02.016454    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxtnp\" (UniqueName: \"kubernetes.io/projected/994e457f-8114-4391-b93c-3ce35edc5543-kube-api-access-wxtnp\") pod \"maas-api-revoke-sub-79805ca9-d9v4z\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:02.094816 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:02.094784    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:02.471859 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:02.471830    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"]
Nov 03 07:54:02.473177 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:54:02.473121    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994e457f_8114_4391_b93c_3ce35edc5543.slice/crio-b8bc40bb7b0b1a053cd82b5dc1f16290c60b1e2dbaeb496e1ceb9f2647189436 WatchSource:0}: Error finding container b8bc40bb7b0b1a053cd82b5dc1f16290c60b1e2dbaeb496e1ceb9f2647189436: Status 404 returned error can't find the container with id b8bc40bb7b0b1a053cd82b5dc1f16290c60b1e2dbaeb496e1ceb9f2647189436
Nov 03 07:54:03.365595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:03.365560    4470 generic.go:356] "Generic (PLEG): container finished" podID="994e457f-8114-4391-b93c-3ce35edc5543" containerID="bf02ee7b8e576326f5f821aeb56f016d5ab4b25c8932e1b15e961e7b71d6ab00" exitCode=0
Nov 03 07:54:03.365999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:03.365597    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z" event={"ID":"994e457f-8114-4391-b93c-3ce35edc5543","Type":"ContainerDied","Data":"bf02ee7b8e576326f5f821aeb56f016d5ab4b25c8932e1b15e961e7b71d6ab00"}
Nov 03 07:54:03.365999 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:03.365658    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z" event={"ID":"994e457f-8114-4391-b93c-3ce35edc5543","Type":"ContainerStarted","Data":"b8bc40bb7b0b1a053cd82b5dc1f16290c60b1e2dbaeb496e1ceb9f2647189436"}
Nov 03 07:54:04.510778 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:04.510752    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:04.578303 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:04.578268    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994e457f-8114-4391-b93c-3ce35edc5543-kube-api-access-wxtnp\" (UniqueName: \"kubernetes.io/projected/994e457f-8114-4391-b93c-3ce35edc5543-kube-api-access-wxtnp\") pod \"994e457f-8114-4391-b93c-3ce35edc5543\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") "
Nov 03 07:54:04.578496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:04.578352    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-xks-ca\") pod \"994e457f-8114-4391-b93c-3ce35edc5543\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") "
Nov 03 07:54:04.578496 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:04.578379    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-openshift-ca\") pod \"994e457f-8114-4391-b93c-3ce35edc5543\" (UID: \"994e457f-8114-4391-b93c-3ce35edc5543\") "
Nov 03 07:54:04.578821 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:04.578795    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-openshift-ca" pod "994e457f-8114-4391-b93c-3ce35edc5543" (UID: "994e457f-8114-4391-b93c-3ce35edc5543"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:04.580585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:04.580564    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994e457f-8114-4391-b93c-3ce35edc5543-kube-api-access-wxtnp" pod "994e457f-8114-4391-b93c-3ce35edc5543" (UID: "994e457f-8114-4391-b93c-3ce35edc5543"). InnerVolumeSpecName "kube-api-access-wxtnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:04.580585 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:04.580577    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-xks-ca" pod "994e457f-8114-4391-b93c-3ce35edc5543" (UID: "994e457f-8114-4391-b93c-3ce35edc5543"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:04.679953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:04.679877    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxtnp\" (UniqueName: \"kubernetes.io/projected/994e457f-8114-4391-b93c-3ce35edc5543-kube-api-access-wxtnp\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:54:04.679953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:04.679906    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:54:04.679953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:04.679915    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e457f-8114-4391-b93c-3ce35edc5543-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:54:04.758409 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:54:04.758376    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:54:05.377443 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:05.377412    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"
Nov 03 07:54:05.377732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:05.377461    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z" event={"ID":"994e457f-8114-4391-b93c-3ce35edc5543","Type":"ContainerDied","Data":"b8bc40bb7b0b1a053cd82b5dc1f16290c60b1e2dbaeb496e1ceb9f2647189436"}
Nov 03 07:54:05.377732 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:05.377516    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8bc40bb7b0b1a053cd82b5dc1f16290c60b1e2dbaeb496e1ceb9f2647189436"
Nov 03 07:54:15.090468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:15.090437    4470 scope.go:122] "RemoveContainer" containerID="11046d221bfec023fed731584bd88a41072ad66ecfa8d9bf98e7c0426e0fe24a"
Nov 03 07:54:15.101836 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:15.101815    4470 scope.go:122] "RemoveContainer" containerID="582bb28008580f93592c6efb9959a62c94e044bd182241895718c7f18a38a314"
Nov 03 07:54:15.110110 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:15.110091    4470 scope.go:122] "RemoveContainer" containerID="1e2294f8d2be32ee7bb0248ddacc0a3736ba5765fbd68e0085af37fce6594711"
Nov 03 07:54:15.118039 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:15.118015    4470 scope.go:122] "RemoveContainer" containerID="005b770539e13a28117be9615eeb9537a18dd5a0e67883128878fcb774b92b5c"
Nov 03 07:54:15.125662 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:15.125641    4470 scope.go:122] "RemoveContainer" containerID="9f00cba410b4d3877f3e9b502e1e4e6d27dbf81462fb5469caadc253c363dd8e"
Nov 03 07:54:15.133632 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:15.133610    4470 scope.go:122] "RemoveContainer" containerID="99ed64ea7f9f9434bcaf4844946958825372110d79d20183646485ba1cc5972f"
Nov 03 07:54:25.585823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.585783    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"]
Nov 03 07:54:25.586328 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.586311    4470 cpu_manager.go:414] "containerMap: removing container" podUID="994e457f-8114-4391-b93c-3ce35edc5543" containerName="revoke-keys"
Nov 03 07:54:25.586388 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.586329    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994e457f-8114-4391-b93c-3ce35edc5543" containerName="revoke-keys"
Nov 03 07:54:25.586436 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.586409    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="994e457f-8114-4391-b93c-3ce35edc5543" containerName="revoke-keys"
Nov 03 07:54:25.590976 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.590960    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:25.650628 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.650593    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"]
Nov 03 07:54:25.655073 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.655042    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nbrz8\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:25.655278 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.655215    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nbrz8\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:25.655278 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.655256    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnz5g\" (UniqueName: \"kubernetes.io/projected/cbe3eb44-fab8-4d9e-809d-4511e2f48120-kube-api-access-wnz5g\") pod \"maas-api-revoke-sub-0b7e8edb-nbrz8\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:25.696803 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.696763    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:54:25.696803 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.696786    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:54:25.697047 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.696900    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:54:25.699945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.699925    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:54:25.755968 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.755935    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nbrz8\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:25.756204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.755974    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnz5g\" (UniqueName: \"kubernetes.io/projected/cbe3eb44-fab8-4d9e-809d-4511e2f48120-kube-api-access-wnz5g\") pod \"maas-api-revoke-sub-0b7e8edb-nbrz8\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:25.756204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.756043    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nbrz8\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:25.756840 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.756815    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nbrz8\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:25.758596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.758575    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nbrz8\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:25.820708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.820672    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnz5g\" (UniqueName: \"kubernetes.io/projected/cbe3eb44-fab8-4d9e-809d-4511e2f48120-kube-api-access-wnz5g\") pod \"maas-api-revoke-sub-0b7e8edb-nbrz8\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:25.903997 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:25.903911    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:26.279250 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:26.279217    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"]
Nov 03 07:54:26.280543 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:54:26.280517    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe3eb44_fab8_4d9e_809d_4511e2f48120.slice/crio-71ccfdcc9d409d7db6b0f30b1395d83111dcb25d13f8910827d1287f413c1bc6 WatchSource:0}: Error finding container 71ccfdcc9d409d7db6b0f30b1395d83111dcb25d13f8910827d1287f413c1bc6: Status 404 returned error can't find the container with id 71ccfdcc9d409d7db6b0f30b1395d83111dcb25d13f8910827d1287f413c1bc6
Nov 03 07:54:26.510650 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:26.510615    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8" event={"ID":"cbe3eb44-fab8-4d9e-809d-4511e2f48120","Type":"ContainerStarted","Data":"71ccfdcc9d409d7db6b0f30b1395d83111dcb25d13f8910827d1287f413c1bc6"}
Nov 03 07:54:27.517975 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:27.517940    4470 generic.go:356] "Generic (PLEG): container finished" podID="cbe3eb44-fab8-4d9e-809d-4511e2f48120" containerID="f1302c4a44776345521380abe65588dbb6464e4b119d26ff37f8d4eb888615d9" exitCode=0
Nov 03 07:54:27.518382 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:27.517988    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8" event={"ID":"cbe3eb44-fab8-4d9e-809d-4511e2f48120","Type":"ContainerDied","Data":"f1302c4a44776345521380abe65588dbb6464e4b119d26ff37f8d4eb888615d9"}
Nov 03 07:54:28.665595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:28.665572    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:28.785485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:28.785320    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-xks-ca\") pod \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") "
Nov 03 07:54:28.785485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:28.785451    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe3eb44-fab8-4d9e-809d-4511e2f48120-kube-api-access-wnz5g\" (UniqueName: \"kubernetes.io/projected/cbe3eb44-fab8-4d9e-809d-4511e2f48120-kube-api-access-wnz5g\") pod \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") "
Nov 03 07:54:28.785733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:28.785487    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-openshift-ca\") pod \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\" (UID: \"cbe3eb44-fab8-4d9e-809d-4511e2f48120\") "
Nov 03 07:54:28.786341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:28.786289    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-openshift-ca" pod "cbe3eb44-fab8-4d9e-809d-4511e2f48120" (UID: "cbe3eb44-fab8-4d9e-809d-4511e2f48120"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:28.788846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:28.788695    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-xks-ca" pod "cbe3eb44-fab8-4d9e-809d-4511e2f48120" (UID: "cbe3eb44-fab8-4d9e-809d-4511e2f48120"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:28.788846 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:28.788714    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe3eb44-fab8-4d9e-809d-4511e2f48120-kube-api-access-wnz5g" pod "cbe3eb44-fab8-4d9e-809d-4511e2f48120" (UID: "cbe3eb44-fab8-4d9e-809d-4511e2f48120"). InnerVolumeSpecName "kube-api-access-wnz5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:28.886956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:28.886920    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:54:28.886956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:28.886951    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnz5g\" (UniqueName: \"kubernetes.io/projected/cbe3eb44-fab8-4d9e-809d-4511e2f48120-kube-api-access-wnz5g\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:54:28.886956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:28.886963    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe3eb44-fab8-4d9e-809d-4511e2f48120-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:54:29.529474 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:29.529445    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"
Nov 03 07:54:29.529677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:29.529490    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8" event={"ID":"cbe3eb44-fab8-4d9e-809d-4511e2f48120","Type":"ContainerDied","Data":"71ccfdcc9d409d7db6b0f30b1395d83111dcb25d13f8910827d1287f413c1bc6"}
Nov 03 07:54:29.529677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:29.529544    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ccfdcc9d409d7db6b0f30b1395d83111dcb25d13f8910827d1287f413c1bc6"
Nov 03 07:54:34.759160 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:54:34.759108    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:54:36.923960 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:36.923920    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"]
Nov 03 07:54:36.924468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:36.924368    4470 cpu_manager.go:414] "containerMap: removing container" podUID="cbe3eb44-fab8-4d9e-809d-4511e2f48120" containerName="revoke-keys"
Nov 03 07:54:36.924468 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:36.924381    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe3eb44-fab8-4d9e-809d-4511e2f48120" containerName="revoke-keys"
Nov 03 07:54:36.924547 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:36.924470    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe3eb44-fab8-4d9e-809d-4511e2f48120" containerName="revoke-keys"
Nov 03 07:54:36.927884 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:36.927868    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:36.986213 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:36.986178    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"]
Nov 03 07:54:37.033875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.033845    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:54:37.033875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.033850    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:54:37.033875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.033860    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:54:37.034167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.033920    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:54:37.084492 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.084449    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzmtg\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:37.084694 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.084601    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vwl\" (UniqueName: \"kubernetes.io/projected/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-kube-api-access-n2vwl\") pod \"maas-api-revoke-sub-3d4156f1-dzmtg\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:37.084694 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.084652    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzmtg\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:37.186199 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.186106    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzmtg\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:37.186364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.186219    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vwl\" (UniqueName: \"kubernetes.io/projected/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-kube-api-access-n2vwl\") pod \"maas-api-revoke-sub-3d4156f1-dzmtg\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:37.186364 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.186257    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzmtg\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:37.186763 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.186743    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzmtg\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:37.188605 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.188582    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzmtg\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:37.250899 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.250868    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vwl\" (UniqueName: \"kubernetes.io/projected/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-kube-api-access-n2vwl\") pod \"maas-api-revoke-sub-3d4156f1-dzmtg\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:37.540434 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.540349    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:37.915059 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:37.915030    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"]
Nov 03 07:54:37.916998 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:54:37.916975    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442bd9b1_aae7_4dc9_8ae5_d4c5fd209e62.slice/crio-cf874312caacdca8e7e3a0363b00e4a29cb52b9ed393c3ffb28c2eaece6f98c3 WatchSource:0}: Error finding container cf874312caacdca8e7e3a0363b00e4a29cb52b9ed393c3ffb28c2eaece6f98c3: Status 404 returned error can't find the container with id cf874312caacdca8e7e3a0363b00e4a29cb52b9ed393c3ffb28c2eaece6f98c3
Nov 03 07:54:38.577630 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:38.577590    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg" event={"ID":"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62","Type":"ContainerStarted","Data":"5cac1b65a0c450256e66841432dae400b30ca26f538a9dac727d8b3c1882a371"}
Nov 03 07:54:38.578052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:38.577651    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg" event={"ID":"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62","Type":"ContainerStarted","Data":"cf874312caacdca8e7e3a0363b00e4a29cb52b9ed393c3ffb28c2eaece6f98c3"}
Nov 03 07:54:38.698835 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:38.698778    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg" podStartSLOduration=2.6987645479999998 podStartE2EDuration="2.698764548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:54:38.69788294 +0000 UTC m=+3331.030251936" watchObservedRunningTime="2026-11-03 07:54:38.698764548 +0000 UTC m=+3331.031133543"
Nov 03 07:54:39.598686 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:39.598650    4470 generic.go:356] "Generic (PLEG): container finished" podID="442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62" containerID="5cac1b65a0c450256e66841432dae400b30ca26f538a9dac727d8b3c1882a371" exitCode=0
Nov 03 07:54:39.604266 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:39.603220    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg" event={"ID":"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62","Type":"ContainerDied","Data":"5cac1b65a0c450256e66841432dae400b30ca26f538a9dac727d8b3c1882a371"}
Nov 03 07:54:40.854295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:40.854268    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:40.880646 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:40.880612    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-xks-ca\") pod \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") "
Nov 03 07:54:40.880814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:40.880706    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-kube-api-access-n2vwl\" (UniqueName: \"kubernetes.io/projected/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-kube-api-access-n2vwl\") pod \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") "
Nov 03 07:54:40.880814 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:40.880782    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-openshift-ca\") pod \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\" (UID: \"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62\") "
Nov 03 07:54:40.881105 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:40.881079    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-openshift-ca" pod "442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62" (UID: "442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:40.882888 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:40.882862    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-xks-ca" pod "442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62" (UID: "442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:40.882990 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:40.882956    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-kube-api-access-n2vwl" pod "442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62" (UID: "442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62"). InnerVolumeSpecName "kube-api-access-n2vwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:40.982158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:40.982113    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:54:40.982158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:40.982155    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vwl\" (UniqueName: \"kubernetes.io/projected/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-kube-api-access-n2vwl\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:54:40.982158 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:40.982166    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:54:41.638090 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:41.638056    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"
Nov 03 07:54:41.638306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:41.638082    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg" event={"ID":"442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62","Type":"ContainerDied","Data":"cf874312caacdca8e7e3a0363b00e4a29cb52b9ed393c3ffb28c2eaece6f98c3"}
Nov 03 07:54:41.638306 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:41.638154    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf874312caacdca8e7e3a0363b00e4a29cb52b9ed393c3ffb28c2eaece6f98c3"
Nov 03 07:54:54.761401 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:54.761337    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 07:54:54.761957 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:54.761426    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" podUID="4dfa6c32-f24d-4d1f-96ff-df222dec2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 07:54:55.061595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:55.061505    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"]
Nov 03 07:54:55.112595 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:55.112555    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ckphc"]
Nov 03 07:54:55.805080 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:54:55.805040    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52cec711-289e-4862-b94c-daa4d2fa0281" path="/var/lib/kubelet/pods/52cec711-289e-4862-b94c-daa4d2fa0281/volumes"
Nov 03 07:55:04.757860 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:55:04.757822    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:55:10.604956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.604919    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"]
Nov 03 07:55:10.655418 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.655385    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzmtg"]
Nov 03 07:55:10.731104 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.731070    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"]
Nov 03 07:55:10.731518 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.731505    4470 cpu_manager.go:414] "containerMap: removing container" podUID="442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62" containerName="revoke-keys"
Nov 03 07:55:10.731570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.731519    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62" containerName="revoke-keys"
Nov 03 07:55:10.731621 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.731611    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62" containerName="revoke-keys"
Nov 03 07:55:10.734781 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.734765    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:10.791649 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.791618    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:55:10.791887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.791739    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:55:10.791887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.791821    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:55:10.792067 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.792029    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:55:10.794553 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.794527    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"]
Nov 03 07:55:10.850273 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.850241    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xqp4\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:10.850475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.850285    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xqp4\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:10.850475 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.850404    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4g9h\" (UniqueName: \"kubernetes.io/projected/beefd4ae-b300-452b-8e77-9e9b604fca58-kube-api-access-l4g9h\") pod \"maas-api-revoke-sub-3d4156f1-5xqp4\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:10.951459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.951380    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xqp4\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:10.951459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.951425    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xqp4\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:10.951459 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.951460    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4g9h\" (UniqueName: \"kubernetes.io/projected/beefd4ae-b300-452b-8e77-9e9b604fca58-kube-api-access-l4g9h\") pod \"maas-api-revoke-sub-3d4156f1-5xqp4\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:10.952016 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.951994    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xqp4\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:10.953944 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:10.953916    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xqp4\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:11.016337 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:11.016306    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4g9h\" (UniqueName: \"kubernetes.io/projected/beefd4ae-b300-452b-8e77-9e9b604fca58-kube-api-access-l4g9h\") pod \"maas-api-revoke-sub-3d4156f1-5xqp4\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:11.047761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:11.047734    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:11.423296 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:11.423260    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"]
Nov 03 07:55:11.424684 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:55:11.424657    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeefd4ae_b300_452b_8e77_9e9b604fca58.slice/crio-6709b1ed7dc41c69619363dfec23c69d659d5fc11ed005a206e5fa11f276f680 WatchSource:0}: Error finding container 6709b1ed7dc41c69619363dfec23c69d659d5fc11ed005a206e5fa11f276f680: Status 404 returned error can't find the container with id 6709b1ed7dc41c69619363dfec23c69d659d5fc11ed005a206e5fa11f276f680
Nov 03 07:55:11.806186 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:11.806148    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62" path="/var/lib/kubelet/pods/442bd9b1-aae7-4dc9-8ae5-d4c5fd209e62/volumes"
Nov 03 07:55:11.844714 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:11.844684    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4" event={"ID":"beefd4ae-b300-452b-8e77-9e9b604fca58","Type":"ContainerStarted","Data":"ccb248af23ba27530e782272169269f42d4e3e265385409ec2d834dc1502a3ba"}
Nov 03 07:55:11.844878 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:11.844737    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4" event={"ID":"beefd4ae-b300-452b-8e77-9e9b604fca58","Type":"ContainerStarted","Data":"6709b1ed7dc41c69619363dfec23c69d659d5fc11ed005a206e5fa11f276f680"}
Nov 03 07:55:11.965337 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:11.965290    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4" podStartSLOduration=1.9652761939999999 podStartE2EDuration="1.965276194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:55:11.964230054 +0000 UTC m=+3364.296599059" watchObservedRunningTime="2026-11-03 07:55:11.965276194 +0000 UTC m=+3364.297645179"
Nov 03 07:55:12.852272 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:12.852243    4470 generic.go:356] "Generic (PLEG): container finished" podID="beefd4ae-b300-452b-8e77-9e9b604fca58" containerID="ccb248af23ba27530e782272169269f42d4e3e265385409ec2d834dc1502a3ba" exitCode=0
Nov 03 07:55:12.852689 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:12.852304    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4" event={"ID":"beefd4ae-b300-452b-8e77-9e9b604fca58","Type":"ContainerDied","Data":"ccb248af23ba27530e782272169269f42d4e3e265385409ec2d834dc1502a3ba"}
Nov 03 07:55:13.995874 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:13.995843    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:14.084159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.084101    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beefd4ae-b300-452b-8e77-9e9b604fca58-kube-api-access-l4g9h\" (UniqueName: \"kubernetes.io/projected/beefd4ae-b300-452b-8e77-9e9b604fca58-kube-api-access-l4g9h\") pod \"beefd4ae-b300-452b-8e77-9e9b604fca58\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") "
Nov 03 07:55:14.084366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.084207    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-xks-ca\") pod \"beefd4ae-b300-452b-8e77-9e9b604fca58\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") "
Nov 03 07:55:14.084366 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.084272    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-openshift-ca\") pod \"beefd4ae-b300-452b-8e77-9e9b604fca58\" (UID: \"beefd4ae-b300-452b-8e77-9e9b604fca58\") "
Nov 03 07:55:14.084616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.084591    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-openshift-ca" pod "beefd4ae-b300-452b-8e77-9e9b604fca58" (UID: "beefd4ae-b300-452b-8e77-9e9b604fca58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:55:14.086375 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.086352    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beefd4ae-b300-452b-8e77-9e9b604fca58-kube-api-access-l4g9h" pod "beefd4ae-b300-452b-8e77-9e9b604fca58" (UID: "beefd4ae-b300-452b-8e77-9e9b604fca58"). InnerVolumeSpecName "kube-api-access-l4g9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:55:14.086487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.086354    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-xks-ca" pod "beefd4ae-b300-452b-8e77-9e9b604fca58" (UID: "beefd4ae-b300-452b-8e77-9e9b604fca58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:55:14.185570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.185483    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:55:14.185570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.185514    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4g9h\" (UniqueName: \"kubernetes.io/projected/beefd4ae-b300-452b-8e77-9e9b604fca58-kube-api-access-l4g9h\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:55:14.185570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.185525    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beefd4ae-b300-452b-8e77-9e9b604fca58-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:55:14.863555 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.863522    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4" event={"ID":"beefd4ae-b300-452b-8e77-9e9b604fca58","Type":"ContainerDied","Data":"6709b1ed7dc41c69619363dfec23c69d659d5fc11ed005a206e5fa11f276f680"}
Nov 03 07:55:14.863747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.863571    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6709b1ed7dc41c69619363dfec23c69d659d5fc11ed005a206e5fa11f276f680"
Nov 03 07:55:14.863747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:14.863538    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"
Nov 03 07:55:15.172431 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:55:15.172353    4470 scope.go:122] "RemoveContainer" containerID="fbf3ffbe9301b0b054f757c9854eff5032d9c5efc446a5f213a14b718b0804e2"
Nov 03 07:55:34.759694 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:55:34.759607    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:56:02.055911 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:02.055877    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"]
Nov 03 07:56:02.108323 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:02.108281    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-x7n92"]
Nov 03 07:56:03.805264 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:03.805232    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4abaca6-d153-428d-8f2a-f2c1628046b9" path="/var/lib/kubelet/pods/d4abaca6-d153-428d-8f2a-f2c1628046b9/volumes"
Nov 03 07:56:04.246894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.246860    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"]
Nov 03 07:56:04.247379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.247359    4470 cpu_manager.go:414] "containerMap: removing container" podUID="beefd4ae-b300-452b-8e77-9e9b604fca58" containerName="revoke-keys"
Nov 03 07:56:04.247379 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.247377    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beefd4ae-b300-452b-8e77-9e9b604fca58" containerName="revoke-keys"
Nov 03 07:56:04.247536 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.247470    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="beefd4ae-b300-452b-8e77-9e9b604fca58" containerName="revoke-keys"
Nov 03 07:56:04.250866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.250850    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:04.310562 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.310525    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"]
Nov 03 07:56:04.334408 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.334191    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5pskn\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:04.334798 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.334773    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5pskn\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:04.334972 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.334956    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jlnc\" (UniqueName: \"kubernetes.io/projected/21540773-a8f7-4bab-9bd5-570a6a90fa78-kube-api-access-8jlnc\") pod \"maas-api-revoke-sub-e2fc9c5b-5pskn\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:04.357570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.357541    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:56:04.357570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.357559    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:56:04.357570 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.357561    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:56:04.357776 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.357624    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:56:04.435682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.435648    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5pskn\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:04.435864 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.435799    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5pskn\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:04.435864 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.435843    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlnc\" (UniqueName: \"kubernetes.io/projected/21540773-a8f7-4bab-9bd5-570a6a90fa78-kube-api-access-8jlnc\") pod \"maas-api-revoke-sub-e2fc9c5b-5pskn\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:04.436444 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.436415    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5pskn\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:04.438341 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.438315    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5pskn\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:04.500818 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.500753    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jlnc\" (UniqueName: \"kubernetes.io/projected/21540773-a8f7-4bab-9bd5-570a6a90fa78-kube-api-access-8jlnc\") pod \"maas-api-revoke-sub-e2fc9c5b-5pskn\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:04.563702 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.563669    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:04.761343 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:56:04.761259    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:56:04.939331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:04.939303    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"]
Nov 03 07:56:04.940548 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:56:04.940523    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21540773_a8f7_4bab_9bd5_570a6a90fa78.slice/crio-2ce685b14f564204d006d5ca55105f3ddc2c2a0bd3c6c57ff28e764df828891a WatchSource:0}: Error finding container 2ce685b14f564204d006d5ca55105f3ddc2c2a0bd3c6c57ff28e764df828891a: Status 404 returned error can't find the container with id 2ce685b14f564204d006d5ca55105f3ddc2c2a0bd3c6c57ff28e764df828891a
Nov 03 07:56:05.157115 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:05.157083    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn" event={"ID":"21540773-a8f7-4bab-9bd5-570a6a90fa78","Type":"ContainerStarted","Data":"2ce685b14f564204d006d5ca55105f3ddc2c2a0bd3c6c57ff28e764df828891a"}
Nov 03 07:56:06.164449 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:06.164418    4470 generic.go:356] "Generic (PLEG): container finished" podID="21540773-a8f7-4bab-9bd5-570a6a90fa78" containerID="4bb599a3ed86e7ec6e02bfa7d79b7686dd6fc5f31d58db45514c5528d6dd6f5e" exitCode=0
Nov 03 07:56:06.164881 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:06.164496    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn" event={"ID":"21540773-a8f7-4bab-9bd5-570a6a90fa78","Type":"ContainerDied","Data":"4bb599a3ed86e7ec6e02bfa7d79b7686dd6fc5f31d58db45514c5528d6dd6f5e"}
Nov 03 07:56:07.313735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:07.313708    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:07.358498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:07.358466    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-xks-ca\") pod \"21540773-a8f7-4bab-9bd5-570a6a90fa78\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") "
Nov 03 07:56:07.358735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:07.358513    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21540773-a8f7-4bab-9bd5-570a6a90fa78-kube-api-access-8jlnc\" (UniqueName: \"kubernetes.io/projected/21540773-a8f7-4bab-9bd5-570a6a90fa78-kube-api-access-8jlnc\") pod \"21540773-a8f7-4bab-9bd5-570a6a90fa78\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") "
Nov 03 07:56:07.358735 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:07.358561    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-openshift-ca\") pod \"21540773-a8f7-4bab-9bd5-570a6a90fa78\" (UID: \"21540773-a8f7-4bab-9bd5-570a6a90fa78\") "
Nov 03 07:56:07.358953 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:07.358925    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-openshift-ca" pod "21540773-a8f7-4bab-9bd5-570a6a90fa78" (UID: "21540773-a8f7-4bab-9bd5-570a6a90fa78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:56:07.360795 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:07.360773    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21540773-a8f7-4bab-9bd5-570a6a90fa78-kube-api-access-8jlnc" pod "21540773-a8f7-4bab-9bd5-570a6a90fa78" (UID: "21540773-a8f7-4bab-9bd5-570a6a90fa78"). InnerVolumeSpecName "kube-api-access-8jlnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:56:07.360875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:07.360806    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-xks-ca" pod "21540773-a8f7-4bab-9bd5-570a6a90fa78" (UID: "21540773-a8f7-4bab-9bd5-570a6a90fa78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:56:07.459514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:07.459429    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:56:07.459514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:07.459463    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jlnc\" (UniqueName: \"kubernetes.io/projected/21540773-a8f7-4bab-9bd5-570a6a90fa78-kube-api-access-8jlnc\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:56:07.459514 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:07.459479    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21540773-a8f7-4bab-9bd5-570a6a90fa78-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:56:08.177162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:08.177063    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"
Nov 03 07:56:08.177162 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:08.177092    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn" event={"ID":"21540773-a8f7-4bab-9bd5-570a6a90fa78","Type":"ContainerDied","Data":"2ce685b14f564204d006d5ca55105f3ddc2c2a0bd3c6c57ff28e764df828891a"}
Nov 03 07:56:08.177356 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:08.177169    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ce685b14f564204d006d5ca55105f3ddc2c2a0bd3c6c57ff28e764df828891a"
Nov 03 07:56:15.011720 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.011685    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"]
Nov 03 07:56:15.012156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.012096    4470 cpu_manager.go:414] "containerMap: removing container" podUID="21540773-a8f7-4bab-9bd5-570a6a90fa78" containerName="revoke-keys"
Nov 03 07:56:15.012156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.012108    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21540773-a8f7-4bab-9bd5-570a6a90fa78" containerName="revoke-keys"
Nov 03 07:56:15.012243 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.012224    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="21540773-a8f7-4bab-9bd5-570a6a90fa78" containerName="revoke-keys"
Nov 03 07:56:15.015389 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.015371    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:15.080081 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.075836    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"]
Nov 03 07:56:15.122568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.122534    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:56:15.122747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.122570    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:56:15.122747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.122571    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:56:15.122747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.122599    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:56:15.127148 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.127119    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2658\" (UniqueName: \"kubernetes.io/projected/c73d0b64-478d-41a5-9256-88dfa0879aa7-kube-api-access-w2658\") pod \"maas-api-revoke-sub-5a29efb3-7vp9z\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:15.127204 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.127194    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7vp9z\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:15.127344 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.127324    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7vp9z\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:15.197106 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.197074    4470 scope.go:122] "RemoveContainer" containerID="147bc04255287a6cc75f5ccb9b8320787eb07333c3e8f8420abea8f890030ab3"
Nov 03 07:56:15.228318 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.228287    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7vp9z\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:15.228451 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.228386    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7vp9z\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:15.228451 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.228422    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2658\" (UniqueName: \"kubernetes.io/projected/c73d0b64-478d-41a5-9256-88dfa0879aa7-kube-api-access-w2658\") pod \"maas-api-revoke-sub-5a29efb3-7vp9z\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:15.228875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.228856    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7vp9z\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:15.230832 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.230801    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7vp9z\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:15.293010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.292944    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2658\" (UniqueName: \"kubernetes.io/projected/c73d0b64-478d-41a5-9256-88dfa0879aa7-kube-api-access-w2658\") pod \"maas-api-revoke-sub-5a29efb3-7vp9z\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:15.328043 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.328014    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:15.702589 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:15.702563    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"]
Nov 03 07:56:15.703988 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:56:15.703959    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73d0b64_478d_41a5_9256_88dfa0879aa7.slice/crio-71a8f385a7226a28186f3053b383609340f8523b7794250b69b275f6e97f8a7a WatchSource:0}: Error finding container 71a8f385a7226a28186f3053b383609340f8523b7794250b69b275f6e97f8a7a: Status 404 returned error can't find the container with id 71a8f385a7226a28186f3053b383609340f8523b7794250b69b275f6e97f8a7a
Nov 03 07:56:16.220522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:16.220430    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z" event={"ID":"c73d0b64-478d-41a5-9256-88dfa0879aa7","Type":"ContainerStarted","Data":"ef1c257d9d123cb3da9e1d8cd228e6d1b39f544486f3c4361903520eff00cd90"}
Nov 03 07:56:16.220902 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:16.220532    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z" event={"ID":"c73d0b64-478d-41a5-9256-88dfa0879aa7","Type":"ContainerStarted","Data":"71a8f385a7226a28186f3053b383609340f8523b7794250b69b275f6e97f8a7a"}
Nov 03 07:56:16.340855 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:16.340799    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z" podStartSLOduration=2.3407864050000002 podStartE2EDuration="2.340786405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:56:16.340216537 +0000 UTC m=+3428.672585531" watchObservedRunningTime="2026-11-03 07:56:16.340786405 +0000 UTC m=+3428.673155399"
Nov 03 07:56:17.226745 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:17.226710    4470 generic.go:356] "Generic (PLEG): container finished" podID="c73d0b64-478d-41a5-9256-88dfa0879aa7" containerID="ef1c257d9d123cb3da9e1d8cd228e6d1b39f544486f3c4361903520eff00cd90" exitCode=0
Nov 03 07:56:17.227193 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:17.226797    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z" event={"ID":"c73d0b64-478d-41a5-9256-88dfa0879aa7","Type":"ContainerDied","Data":"ef1c257d9d123cb3da9e1d8cd228e6d1b39f544486f3c4361903520eff00cd90"}
Nov 03 07:56:18.459976 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:18.459912    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:18.573173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:18.566519    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-xks-ca\") pod \"c73d0b64-478d-41a5-9256-88dfa0879aa7\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") "
Nov 03 07:56:18.573173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:18.566627    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-openshift-ca\") pod \"c73d0b64-478d-41a5-9256-88dfa0879aa7\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") "
Nov 03 07:56:18.573173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:18.566733    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73d0b64-478d-41a5-9256-88dfa0879aa7-kube-api-access-w2658\" (UniqueName: \"kubernetes.io/projected/c73d0b64-478d-41a5-9256-88dfa0879aa7-kube-api-access-w2658\") pod \"c73d0b64-478d-41a5-9256-88dfa0879aa7\" (UID: \"c73d0b64-478d-41a5-9256-88dfa0879aa7\") "
Nov 03 07:56:18.573173 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:18.567827    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-openshift-ca" pod "c73d0b64-478d-41a5-9256-88dfa0879aa7" (UID: "c73d0b64-478d-41a5-9256-88dfa0879aa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:56:18.588156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:18.586687    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-xks-ca" pod "c73d0b64-478d-41a5-9256-88dfa0879aa7" (UID: "c73d0b64-478d-41a5-9256-88dfa0879aa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:56:18.588156 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:18.586844    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73d0b64-478d-41a5-9256-88dfa0879aa7-kube-api-access-w2658" pod "c73d0b64-478d-41a5-9256-88dfa0879aa7" (UID: "c73d0b64-478d-41a5-9256-88dfa0879aa7"). InnerVolumeSpecName "kube-api-access-w2658". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:56:18.676159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:18.675853    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:56:18.676159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:18.675894    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73d0b64-478d-41a5-9256-88dfa0879aa7-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:56:18.676159 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:18.675913    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2658\" (UniqueName: \"kubernetes.io/projected/c73d0b64-478d-41a5-9256-88dfa0879aa7-kube-api-access-w2658\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:56:19.263166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:19.259240    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z" event={"ID":"c73d0b64-478d-41a5-9256-88dfa0879aa7","Type":"ContainerDied","Data":"71a8f385a7226a28186f3053b383609340f8523b7794250b69b275f6e97f8a7a"}
Nov 03 07:56:19.263166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:19.259306    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71a8f385a7226a28186f3053b383609340f8523b7794250b69b275f6e97f8a7a"
Nov 03 07:56:19.263166 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:19.259400    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"
Nov 03 07:56:33.055565 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:33.055532    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"]
Nov 03 07:56:33.106218 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:33.106189    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-djjnm"]
Nov 03 07:56:33.805825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:33.805790    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad1392a-cf6e-4aa0-b451-6355a498b071" path="/var/lib/kubelet/pods/4ad1392a-cf6e-4aa0-b451-6355a498b071/volumes"
Nov 03 07:56:33.901580 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:33.901543    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"]
Nov 03 07:56:33.951898 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:33.951864    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xqp4"]
Nov 03 07:56:34.027061 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.027020    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"]
Nov 03 07:56:34.027568 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.027548    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c73d0b64-478d-41a5-9256-88dfa0879aa7" containerName="revoke-keys"
Nov 03 07:56:34.027677 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.027570    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73d0b64-478d-41a5-9256-88dfa0879aa7" containerName="revoke-keys"
Nov 03 07:56:34.027736 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.027705    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73d0b64-478d-41a5-9256-88dfa0879aa7" containerName="revoke-keys"
Nov 03 07:56:34.033161 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.032465    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:34.087937 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.087906    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:56:34.088535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.088165    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:56:34.088535 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.088400    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:56:34.090331 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.090306    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"]
Nov 03 07:56:34.137604 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.137578    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:56:34.164823 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.164792    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2rb48\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:34.165004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.164904    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8hf\" (UniqueName: \"kubernetes.io/projected/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-kube-api-access-nj8hf\") pod \"maas-api-revoke-sub-3d4156f1-2rb48\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:34.165004 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.164960    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2rb48\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:34.266410 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.266368    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2rb48\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:34.266605 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.266440    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8hf\" (UniqueName: \"kubernetes.io/projected/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-kube-api-access-nj8hf\") pod \"maas-api-revoke-sub-3d4156f1-2rb48\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:34.266605 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.266542    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2rb48\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:34.267020 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.266998    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2rb48\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:34.269108 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.269087    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2rb48\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:34.335981 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.335944    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8hf\" (UniqueName: \"kubernetes.io/projected/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-kube-api-access-nj8hf\") pod \"maas-api-revoke-sub-3d4156f1-2rb48\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:34.348871 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.348805    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:34.724117 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:34.724090    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"]
Nov 03 07:56:34.726031 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:56:34.725999    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0788bd_f6c4_42f4_b942_43db5ffaedcf.slice/crio-51e7f9adb32a6befd131dfdca37e9c653ac20a4da1969d4c3920f954bb795a81 WatchSource:0}: Error finding container 51e7f9adb32a6befd131dfdca37e9c653ac20a4da1969d4c3920f954bb795a81: Status 404 returned error can't find the container with id 51e7f9adb32a6befd131dfdca37e9c653ac20a4da1969d4c3920f954bb795a81
Nov 03 07:56:34.759456 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:56:34.759427    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:56:35.346610 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:35.346561    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48" event={"ID":"ed0788bd-f6c4-42f4-b942-43db5ffaedcf","Type":"ContainerStarted","Data":"a4ff4c859bc36cbcf2efb293d736dd95a37df412fcb7f705d5502caf3bf713c3"}
Nov 03 07:56:35.347015 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:35.346631    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48" event={"ID":"ed0788bd-f6c4-42f4-b942-43db5ffaedcf","Type":"ContainerStarted","Data":"51e7f9adb32a6befd131dfdca37e9c653ac20a4da1969d4c3920f954bb795a81"}
Nov 03 07:56:35.518157 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:35.518077    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48" podStartSLOduration=2.51806411 podStartE2EDuration="2.51806411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:56:35.467332954 +0000 UTC m=+3447.799701960" watchObservedRunningTime="2026-11-03 07:56:35.51806411 +0000 UTC m=+3447.850433104"
Nov 03 07:56:35.805179 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:35.805148    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beefd4ae-b300-452b-8e77-9e9b604fca58" path="/var/lib/kubelet/pods/beefd4ae-b300-452b-8e77-9e9b604fca58/volumes"
Nov 03 07:56:36.353478 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:36.353442    4470 generic.go:356] "Generic (PLEG): container finished" podID="ed0788bd-f6c4-42f4-b942-43db5ffaedcf" containerID="a4ff4c859bc36cbcf2efb293d736dd95a37df412fcb7f705d5502caf3bf713c3" exitCode=0
Nov 03 07:56:36.353889 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:36.353526    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48" event={"ID":"ed0788bd-f6c4-42f4-b942-43db5ffaedcf","Type":"ContainerDied","Data":"a4ff4c859bc36cbcf2efb293d736dd95a37df412fcb7f705d5502caf3bf713c3"}
Nov 03 07:56:37.498127 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:37.498101    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:37.600064 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:37.600031    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-kube-api-access-nj8hf\" (UniqueName: \"kubernetes.io/projected/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-kube-api-access-nj8hf\") pod \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") "
Nov 03 07:56:37.600308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:37.600072    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-openshift-ca\") pod \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") "
Nov 03 07:56:37.600308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:37.600162    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-xks-ca\") pod \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\" (UID: \"ed0788bd-f6c4-42f4-b942-43db5ffaedcf\") "
Nov 03 07:56:37.600513 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:37.600490    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-openshift-ca" pod "ed0788bd-f6c4-42f4-b942-43db5ffaedcf" (UID: "ed0788bd-f6c4-42f4-b942-43db5ffaedcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:56:37.602450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:37.602423    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-xks-ca" pod "ed0788bd-f6c4-42f4-b942-43db5ffaedcf" (UID: "ed0788bd-f6c4-42f4-b942-43db5ffaedcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:56:37.602450 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:37.602439    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-kube-api-access-nj8hf" pod "ed0788bd-f6c4-42f4-b942-43db5ffaedcf" (UID: "ed0788bd-f6c4-42f4-b942-43db5ffaedcf"). InnerVolumeSpecName "kube-api-access-nj8hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:56:37.701479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:37.701388    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8hf\" (UniqueName: \"kubernetes.io/projected/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-kube-api-access-nj8hf\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:56:37.701479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:37.701423    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:56:37.701479 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:37.701435    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0788bd-f6c4-42f4-b942-43db5ffaedcf-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:56:38.366235 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:38.366196    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48" event={"ID":"ed0788bd-f6c4-42f4-b942-43db5ffaedcf","Type":"ContainerDied","Data":"51e7f9adb32a6befd131dfdca37e9c653ac20a4da1969d4c3920f954bb795a81"}
Nov 03 07:56:38.366235 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:38.366226    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"
Nov 03 07:56:38.366235 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:38.366245    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e7f9adb32a6befd131dfdca37e9c653ac20a4da1969d4c3920f954bb795a81"
Nov 03 07:56:45.060861 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:45.060827    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"]
Nov 03 07:56:45.111407 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:45.111375    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jj6gg"]
Nov 03 07:56:45.810295 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:56:45.810258    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9f0349-0a60-4cd4-9bcd-3f595d8b574e" path="/var/lib/kubelet/pods/0f9f0349-0a60-4cd4-9bcd-3f595d8b574e/volumes"
Nov 03 07:57:04.759627 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:57:04.759583    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:57:14.250038 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.250000    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"]
Nov 03 07:57:14.250579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.250468    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ed0788bd-f6c4-42f4-b942-43db5ffaedcf" containerName="revoke-keys"
Nov 03 07:57:14.250579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.250482    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0788bd-f6c4-42f4-b942-43db5ffaedcf" containerName="revoke-keys"
Nov 03 07:57:14.250579 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.250556    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0788bd-f6c4-42f4-b942-43db5ffaedcf" containerName="revoke-keys"
Nov 03 07:57:14.253754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.253730    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:57:14.312841 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.312812    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"]
Nov 03 07:57:14.360101 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.360067    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-98d4f3\""
Nov 03 07:57:14.360279 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.360178    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a3e9eb47-dockercfg-vch9s\""
Nov 03 07:57:14.393342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.393308    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr6zt\" (UniqueName: \"kubernetes.io/projected/6cfde453-d033-4e52-89b1-937cbd991e3c-kube-api-access-lr6zt\") pod \"payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr\" (UID: \"6cfde453-d033-4e52-89b1-937cbd991e3c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:57:14.393515 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.393441    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cfde453-d033-4e52-89b1-937cbd991e3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr\" (UID: \"6cfde453-d033-4e52-89b1-937cbd991e3c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:57:14.453744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.453707    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"]
Nov 03 07:57:14.457365 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.457348    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:57:14.494312 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.494277    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cfde453-d033-4e52-89b1-937cbd991e3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr\" (UID: \"6cfde453-d033-4e52-89b1-937cbd991e3c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:57:14.494521 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.494382    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr6zt\" (UniqueName: \"kubernetes.io/projected/6cfde453-d033-4e52-89b1-937cbd991e3c-kube-api-access-lr6zt\") pod \"payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr\" (UID: \"6cfde453-d033-4e52-89b1-937cbd991e3c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:57:14.495006 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.494980    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cfde453-d033-4e52-89b1-937cbd991e3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr\" (UID: \"6cfde453-d033-4e52-89b1-937cbd991e3c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:57:14.515887 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.515817    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"]
Nov 03 07:57:14.560622 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.560591    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr6zt\" (UniqueName: \"kubernetes.io/projected/6cfde453-d033-4e52-89b1-937cbd991e3c-kube-api-access-lr6zt\") pod \"payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr\" (UID: \"6cfde453-d033-4e52-89b1-937cbd991e3c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:57:14.572360 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.572332    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:57:14.595713 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.595680    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpctj\" (UniqueName: \"kubernetes.io/projected/14a629b9-0bbc-4906-a654-747ac6e5eae1-kube-api-access-qpctj\") pod \"payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d\" (UID: \"14a629b9-0bbc-4906-a654-747ac6e5eae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:57:14.595849 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.595727    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a629b9-0bbc-4906-a654-747ac6e5eae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d\" (UID: \"14a629b9-0bbc-4906-a654-747ac6e5eae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:57:14.697342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.697300    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpctj\" (UniqueName: \"kubernetes.io/projected/14a629b9-0bbc-4906-a654-747ac6e5eae1-kube-api-access-qpctj\") pod \"payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d\" (UID: \"14a629b9-0bbc-4906-a654-747ac6e5eae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:57:14.697342 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.697348    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a629b9-0bbc-4906-a654-747ac6e5eae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d\" (UID: \"14a629b9-0bbc-4906-a654-747ac6e5eae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:57:14.698006 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.697987    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a629b9-0bbc-4906-a654-747ac6e5eae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d\" (UID: \"14a629b9-0bbc-4906-a654-747ac6e5eae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:57:14.762639 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.762597    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpctj\" (UniqueName: \"kubernetes.io/projected/14a629b9-0bbc-4906-a654-747ac6e5eae1-kube-api-access-qpctj\") pod \"payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d\" (UID: \"14a629b9-0bbc-4906-a654-747ac6e5eae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:57:14.771500 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.771438    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:57:14.956606 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:14.956572    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"]
Nov 03 07:57:14.957900 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:57:14.957872    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cfde453_d033_4e52_89b1_937cbd991e3c.slice/crio-091b963f9cf82a85daed3a99b94787a2c171653c717047b795935124120aa5cd WatchSource:0}: Error finding container 091b963f9cf82a85daed3a99b94787a2c171653c717047b795935124120aa5cd: Status 404 returned error can't find the container with id 091b963f9cf82a85daed3a99b94787a2c171653c717047b795935124120aa5cd
Nov 03 07:57:15.162777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:15.159507    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"]
Nov 03 07:57:15.162777 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:57:15.161777    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a629b9_0bbc_4906_a654_747ac6e5eae1.slice/crio-0800102f702fd6e1be44d839d99af4fd88dbea30c46a729af5ab4b4e9f2abc38 WatchSource:0}: Error finding container 0800102f702fd6e1be44d839d99af4fd88dbea30c46a729af5ab4b4e9f2abc38: Status 404 returned error can't find the container with id 0800102f702fd6e1be44d839d99af4fd88dbea30c46a729af5ab4b4e9f2abc38
Nov 03 07:57:15.218606 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:15.218582    4470 scope.go:122] "RemoveContainer" containerID="3be77d8f1df15a6ed67a10a6af10cb36c3efa98c077547d597bca7fcd6eb9d4f"
Nov 03 07:57:15.569371 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:15.569337    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d" event={"ID":"14a629b9-0bbc-4906-a654-747ac6e5eae1","Type":"ContainerStarted","Data":"13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6"}
Nov 03 07:57:15.569825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:15.569399    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d" event={"ID":"14a629b9-0bbc-4906-a654-747ac6e5eae1","Type":"ContainerStarted","Data":"0800102f702fd6e1be44d839d99af4fd88dbea30c46a729af5ab4b4e9f2abc38"}
Nov 03 07:57:15.569825 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:15.569453    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:57:15.571216 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:15.571186    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr" event={"ID":"6cfde453-d033-4e52-89b1-937cbd991e3c","Type":"ContainerStarted","Data":"3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df"}
Nov 03 07:57:15.571392 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:15.571369    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr" event={"ID":"6cfde453-d033-4e52-89b1-937cbd991e3c","Type":"ContainerStarted","Data":"091b963f9cf82a85daed3a99b94787a2c171653c717047b795935124120aa5cd"}
Nov 03 07:57:15.571480 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:15.571413    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:57:15.689761 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:15.689714    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d" podStartSLOduration=1.689699911 podStartE2EDuration="1.689699911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:57:15.689353373 +0000 UTC m=+3488.021722368" watchObservedRunningTime="2026-11-03 07:57:15.689699911 +0000 UTC m=+3488.022068905"
Nov 03 07:57:15.810031 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:15.809983    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr" podStartSLOduration=1.8099686670000001 podStartE2EDuration="1.809968667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:57:15.808605314 +0000 UTC m=+3488.140974309" watchObservedRunningTime="2026-11-03 07:57:15.809968667 +0000 UTC m=+3488.142337661"
Nov 03 07:57:21.055671 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:21.055632    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"]
Nov 03 07:57:21.105962 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:21.105929    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bzn5w"]
Nov 03 07:57:21.805949 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:21.805916    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d66d55a6-c31d-4a2a-952d-3873e5f7e2ad" path="/var/lib/kubelet/pods/d66d55a6-c31d-4a2a-952d-3873e5f7e2ad/volumes"
Nov 03 07:57:26.579866 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:26.579836    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:57:26.580322 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:26.579916    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:57:32.055394 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:32.055350    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"]
Nov 03 07:57:32.106212 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:32.106182    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wwpkk"]
Nov 03 07:57:33.805668 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:57:33.805622    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91bb7baf-fbe2-4106-9296-7294cdaa1e43" path="/var/lib/kubelet/pods/91bb7baf-fbe2-4106-9296-7294cdaa1e43/volumes"
Nov 03 07:57:34.758622 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:57:34.758585    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:58:04.758854 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:58:04.758821    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:58:07.058670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:07.058627    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"]
Nov 03 07:58:07.109564 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:07.109524    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nbpz9"]
Nov 03 07:58:07.806169 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:07.806120    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a" path="/var/lib/kubelet/pods/a9e0ef2f-0226-4f9a-98a5-f4c49a43b80a/volumes"
Nov 03 07:58:15.241124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:15.241094    4470 scope.go:122] "RemoveContainer" containerID="49f61781ecf78b16cc0a66614343865fbad14190db568239dbfc839c894c6129"
Nov 03 07:58:15.250123 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:15.250105    4470 scope.go:122] "RemoveContainer" containerID="9582816c391f5c5635b39c0d00d274089337b8229c656e4d74a604da4ad17153"
Nov 03 07:58:23.414747 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.414668    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"]
Nov 03 07:58:23.419985 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.419968    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:23.478596 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.478565    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"]
Nov 03 07:58:23.484701 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.484670    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f8ee8b5-cb782\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:23.484869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.484730    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f8ee8b5-cb782\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:23.484869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.484808    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4n2w\" (UniqueName: \"kubernetes.io/projected/0ff340ce-5865-4678-8439-aedb28246f6b-kube-api-access-l4n2w\") pod \"maas-api-revoke-sub-8f8ee8b5-cb782\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:23.525416 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.525367    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:58:23.525598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.525426    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:58:23.525598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.525436    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:58:23.525598 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.525436    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:58:23.585666 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.585631    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f8ee8b5-cb782\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:23.585863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.585681    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f8ee8b5-cb782\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:23.585863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.585703    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4n2w\" (UniqueName: \"kubernetes.io/projected/0ff340ce-5865-4678-8439-aedb28246f6b-kube-api-access-l4n2w\") pod \"maas-api-revoke-sub-8f8ee8b5-cb782\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:23.586415 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.586392    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f8ee8b5-cb782\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:23.588430 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.588408    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f8ee8b5-cb782\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:23.650056 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.650025    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4n2w\" (UniqueName: \"kubernetes.io/projected/0ff340ce-5865-4678-8439-aedb28246f6b-kube-api-access-l4n2w\") pod \"maas-api-revoke-sub-8f8ee8b5-cb782\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:23.733082 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:23.732998    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:24.111055 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:24.111028    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"]
Nov 03 07:58:24.112715 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:58:24.112689    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff340ce_5865_4678_8439_aedb28246f6b.slice/crio-53267b97ff618a852b3b0f26a8d7bf5b00c335eea612fdd397c99e2307c7da01 WatchSource:0}: Error finding container 53267b97ff618a852b3b0f26a8d7bf5b00c335eea612fdd397c99e2307c7da01: Status 404 returned error can't find the container with id 53267b97ff618a852b3b0f26a8d7bf5b00c335eea612fdd397c99e2307c7da01
Nov 03 07:58:24.942167 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:24.942099    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782" event={"ID":"0ff340ce-5865-4678-8439-aedb28246f6b","Type":"ContainerStarted","Data":"b2d9aff1ec30ee6d5dba809318d56dff525094e5c1659687b2391c3e92afe506"}
Nov 03 07:58:24.942575 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:24.942179    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782" event={"ID":"0ff340ce-5865-4678-8439-aedb28246f6b","Type":"ContainerStarted","Data":"53267b97ff618a852b3b0f26a8d7bf5b00c335eea612fdd397c99e2307c7da01"}
Nov 03 07:58:25.062601 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:25.062555    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782" podStartSLOduration=2.0625405470000002 podStartE2EDuration="2.062540547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:25.062082535 +0000 UTC m=+3557.394451549" watchObservedRunningTime="2026-11-03 07:58:25.062540547 +0000 UTC m=+3557.394909560"
Nov 03 07:58:25.949076 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:25.949039    4470 generic.go:356] "Generic (PLEG): container finished" podID="0ff340ce-5865-4678-8439-aedb28246f6b" containerID="b2d9aff1ec30ee6d5dba809318d56dff525094e5c1659687b2391c3e92afe506" exitCode=0
Nov 03 07:58:25.949505 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:25.949112    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782" event={"ID":"0ff340ce-5865-4678-8439-aedb28246f6b","Type":"ContainerDied","Data":"b2d9aff1ec30ee6d5dba809318d56dff525094e5c1659687b2391c3e92afe506"}
Nov 03 07:58:27.085386 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.085363    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:27.119584 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.119557    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff340ce-5865-4678-8439-aedb28246f6b-kube-api-access-l4n2w\" (UniqueName: \"kubernetes.io/projected/0ff340ce-5865-4678-8439-aedb28246f6b-kube-api-access-l4n2w\") pod \"0ff340ce-5865-4678-8439-aedb28246f6b\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") "
Nov 03 07:58:27.119722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.119601    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-openshift-ca\") pod \"0ff340ce-5865-4678-8439-aedb28246f6b\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") "
Nov 03 07:58:27.119722 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.119647    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-xks-ca\") pod \"0ff340ce-5865-4678-8439-aedb28246f6b\" (UID: \"0ff340ce-5865-4678-8439-aedb28246f6b\") "
Nov 03 07:58:27.119988 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.119967    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-openshift-ca" pod "0ff340ce-5865-4678-8439-aedb28246f6b" (UID: "0ff340ce-5865-4678-8439-aedb28246f6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:58:27.121863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.121840    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-xks-ca" pod "0ff340ce-5865-4678-8439-aedb28246f6b" (UID: "0ff340ce-5865-4678-8439-aedb28246f6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:58:27.121931 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.121895    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff340ce-5865-4678-8439-aedb28246f6b-kube-api-access-l4n2w" pod "0ff340ce-5865-4678-8439-aedb28246f6b" (UID: "0ff340ce-5865-4678-8439-aedb28246f6b"). InnerVolumeSpecName "kube-api-access-l4n2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:58:27.220754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.220678    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4n2w\" (UniqueName: \"kubernetes.io/projected/0ff340ce-5865-4678-8439-aedb28246f6b-kube-api-access-l4n2w\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:58:27.220754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.220708    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:58:27.220754 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.220722    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff340ce-5865-4678-8439-aedb28246f6b-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:58:27.960764 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.960732    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782" event={"ID":"0ff340ce-5865-4678-8439-aedb28246f6b","Type":"ContainerDied","Data":"53267b97ff618a852b3b0f26a8d7bf5b00c335eea612fdd397c99e2307c7da01"}
Nov 03 07:58:27.960946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.960786    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53267b97ff618a852b3b0f26a8d7bf5b00c335eea612fdd397c99e2307c7da01"
Nov 03 07:58:27.960946 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:27.960742    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"
Nov 03 07:58:34.758998 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:58:34.758940    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:58:35.058620 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.058541    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"]
Nov 03 07:58:35.109124 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.109091    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-slz5k"]
Nov 03 07:58:35.680324 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.680293    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"]
Nov 03 07:58:35.680708 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.680696    4470 cpu_manager.go:414] "containerMap: removing container" podUID="0ff340ce-5865-4678-8439-aedb28246f6b" containerName="revoke-keys"
Nov 03 07:58:35.680752 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.680710    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff340ce-5865-4678-8439-aedb28246f6b" containerName="revoke-keys"
Nov 03 07:58:35.680806 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.680797    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff340ce-5865-4678-8439-aedb28246f6b" containerName="revoke-keys"
Nov 03 07:58:35.684107 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.684092    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:35.740894 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.740821    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:58:35.741086 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.740869    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:58:35.741208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.740912    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:58:35.741208 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.740960    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:58:35.743359 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.743337    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"]
Nov 03 07:58:35.801573 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.801543    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:35.801956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.801696    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:35.801956 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.801735    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfrg\" (UniqueName: \"kubernetes.io/projected/28a89fa3-b448-4214-8e50-05cd1179b0fc-kube-api-access-zpfrg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:35.808118 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.807021    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0" path="/var/lib/kubelet/pods/9b4db0cf-62c7-4c0e-b1b6-9db15eb033e0/volumes"
Nov 03 07:58:35.903477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.903444    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:35.903690 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.903493    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfrg\" (UniqueName: \"kubernetes.io/projected/28a89fa3-b448-4214-8e50-05cd1179b0fc-kube-api-access-zpfrg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:35.903690 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.903610    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:35.904215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.904181    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:35.906052 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.906032    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:35.967828 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.967798    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfrg\" (UniqueName: \"kubernetes.io/projected/28a89fa3-b448-4214-8e50-05cd1179b0fc-kube-api-access-zpfrg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:35.997494 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:35.997442    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:36.373756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:36.373732    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"]
Nov 03 07:58:36.378656 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:58:36.378348    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a89fa3_b448_4214_8e50_05cd1179b0fc.slice/crio-f4a7226ebe6872eec45509287f0fefeafa442fe9ea22f5b429093ee236830387 WatchSource:0}: Error finding container f4a7226ebe6872eec45509287f0fefeafa442fe9ea22f5b429093ee236830387: Status 404 returned error can't find the container with id f4a7226ebe6872eec45509287f0fefeafa442fe9ea22f5b429093ee236830387
Nov 03 07:58:37.007966 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:37.007874    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8" event={"ID":"28a89fa3-b448-4214-8e50-05cd1179b0fc","Type":"ContainerStarted","Data":"898664b5c9b2561ef2613ad8aec5011be28e894a1241fd68dd7ca0de61e5378c"}
Nov 03 07:58:37.007966 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:37.007937    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8" event={"ID":"28a89fa3-b448-4214-8e50-05cd1179b0fc","Type":"ContainerStarted","Data":"f4a7226ebe6872eec45509287f0fefeafa442fe9ea22f5b429093ee236830387"}
Nov 03 07:58:37.127480 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:37.127432    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8" podStartSLOduration=2.127417511 podStartE2EDuration="2.127417511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:37.127164434 +0000 UTC m=+3569.459533429" watchObservedRunningTime="2026-11-03 07:58:37.127417511 +0000 UTC m=+3569.459786504"
Nov 03 07:58:38.014768 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:38.014734    4470 generic.go:356] "Generic (PLEG): container finished" podID="28a89fa3-b448-4214-8e50-05cd1179b0fc" containerID="898664b5c9b2561ef2613ad8aec5011be28e894a1241fd68dd7ca0de61e5378c" exitCode=0
Nov 03 07:58:38.015311 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:38.014830    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8" event={"ID":"28a89fa3-b448-4214-8e50-05cd1179b0fc","Type":"ContainerDied","Data":"898664b5c9b2561ef2613ad8aec5011be28e894a1241fd68dd7ca0de61e5378c"}
Nov 03 07:58:39.162875 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:39.162854    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:39.238123 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:39.238091    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a89fa3-b448-4214-8e50-05cd1179b0fc-kube-api-access-zpfrg\" (UniqueName: \"kubernetes.io/projected/28a89fa3-b448-4214-8e50-05cd1179b0fc-kube-api-access-zpfrg\") pod \"28a89fa3-b448-4214-8e50-05cd1179b0fc\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") "
Nov 03 07:58:39.238329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:39.238208    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-openshift-ca\") pod \"28a89fa3-b448-4214-8e50-05cd1179b0fc\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") "
Nov 03 07:58:39.238329 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:39.238251    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-xks-ca\") pod \"28a89fa3-b448-4214-8e50-05cd1179b0fc\" (UID: \"28a89fa3-b448-4214-8e50-05cd1179b0fc\") "
Nov 03 07:58:39.238602 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:39.238578    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-openshift-ca" pod "28a89fa3-b448-4214-8e50-05cd1179b0fc" (UID: "28a89fa3-b448-4214-8e50-05cd1179b0fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:58:39.240398 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:39.240372    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-xks-ca" pod "28a89fa3-b448-4214-8e50-05cd1179b0fc" (UID: "28a89fa3-b448-4214-8e50-05cd1179b0fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:58:39.240509 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:39.240408    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a89fa3-b448-4214-8e50-05cd1179b0fc-kube-api-access-zpfrg" pod "28a89fa3-b448-4214-8e50-05cd1179b0fc" (UID: "28a89fa3-b448-4214-8e50-05cd1179b0fc"). InnerVolumeSpecName "kube-api-access-zpfrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:58:39.339945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:39.339912    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpfrg\" (UniqueName: \"kubernetes.io/projected/28a89fa3-b448-4214-8e50-05cd1179b0fc-kube-api-access-zpfrg\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:58:39.339945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:39.339941    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:58:39.339945 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:39.339951    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a89fa3-b448-4214-8e50-05cd1179b0fc-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:58:40.026959 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:40.026918    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8" event={"ID":"28a89fa3-b448-4214-8e50-05cd1179b0fc","Type":"ContainerDied","Data":"f4a7226ebe6872eec45509287f0fefeafa442fe9ea22f5b429093ee236830387"}
Nov 03 07:58:40.027164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:40.026974    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4a7226ebe6872eec45509287f0fefeafa442fe9ea22f5b429093ee236830387"
Nov 03 07:58:40.027164 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:40.026937    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"
Nov 03 07:58:47.205390 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:47.205358    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"]
Nov 03 07:58:47.315178 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:47.315128    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"]
Nov 03 07:58:52.205670 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:52.205596    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr" podUID="6cfde453-d033-4e52-89b1-937cbd991e3c" containerName="payload-processing" containerID="cri-o://3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df" gracePeriod=25
Nov 03 07:58:52.268790 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:52.268757    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"]
Nov 03 07:58:52.315477 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:52.315431    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d" podUID="14a629b9-0bbc-4906-a654-747ac6e5eae1" containerName="payload-pre-processing" containerID="cri-o://13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6" gracePeriod=25
Nov 03 07:58:52.319610 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:52.319571    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98d4f3-mrmj8"]
Nov 03 07:58:53.805215 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:53.805186    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a89fa3-b448-4214-8e50-05cd1179b0fc" path="/var/lib/kubelet/pods/28a89fa3-b448-4214-8e50-05cd1179b0fc/volumes"
Nov 03 07:58:56.578376 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:56.578336    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d" podUID="14a629b9-0bbc-4906-a654-747ac6e5eae1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:58:56.578777 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:58:56.578402    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr" podUID="6cfde453-d033-4e52-89b1-937cbd991e3c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:59:01.261869 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.261832    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"]
Nov 03 07:59:01.311260 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.311227    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2rb48"]
Nov 03 07:59:01.384627 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.384590    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"]
Nov 03 07:59:01.385114 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.385095    4470 cpu_manager.go:414] "containerMap: removing container" podUID="28a89fa3-b448-4214-8e50-05cd1179b0fc" containerName="revoke-keys"
Nov 03 07:59:01.385244 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.385116    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a89fa3-b448-4214-8e50-05cd1179b0fc" containerName="revoke-keys"
Nov 03 07:59:01.385305 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.385258    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a89fa3-b448-4214-8e50-05cd1179b0fc" containerName="revoke-keys"
Nov 03 07:59:01.388715 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.388696    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:01.445447 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.445422    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:59:01.445682 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.445443    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 07:59:01.445756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.445513    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:59:01.445756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.445595    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 07:59:01.447451 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.447427    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"]
Nov 03 07:59:01.459549 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.459523    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5tlw\" (UniqueName: \"kubernetes.io/projected/b0f347c3-e6a6-4ffe-89d4-668f90243806-kube-api-access-r5tlw\") pod \"maas-api-revoke-sub-3d4156f1-5hjk9\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:01.459685 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.459571    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5hjk9\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:01.459727 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.459708    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5hjk9\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:01.560461 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.560375    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5tlw\" (UniqueName: \"kubernetes.io/projected/b0f347c3-e6a6-4ffe-89d4-668f90243806-kube-api-access-r5tlw\") pod \"maas-api-revoke-sub-3d4156f1-5hjk9\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:01.560461 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.560417    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5hjk9\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:01.560716 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.560469    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5hjk9\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:01.561186 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.561158    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5hjk9\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:01.562858 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.562838    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5hjk9\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:01.624168 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.624116    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5tlw\" (UniqueName: \"kubernetes.io/projected/b0f347c3-e6a6-4ffe-89d4-668f90243806-kube-api-access-r5tlw\") pod \"maas-api-revoke-sub-3d4156f1-5hjk9\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:01.701294 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.701256    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:01.806833 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:01.806792    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0788bd-f6c4-42f4-b942-43db5ffaedcf" path="/var/lib/kubelet/pods/ed0788bd-f6c4-42f4-b942-43db5ffaedcf/volumes"
Nov 03 07:59:02.076697 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:02.076673    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"]
Nov 03 07:59:02.078105 ip-10-0-129-24 kubenswrapper[4470]: W1103 07:59:02.078082    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f347c3_e6a6_4ffe_89d4_668f90243806.slice/crio-4bd99f104253431eab763d382262af8a238cef501d35fbf6f07d84480c320359 WatchSource:0}: Error finding container 4bd99f104253431eab763d382262af8a238cef501d35fbf6f07d84480c320359: Status 404 returned error can't find the container with id 4bd99f104253431eab763d382262af8a238cef501d35fbf6f07d84480c320359
Nov 03 07:59:02.152044 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:02.152015    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9" event={"ID":"b0f347c3-e6a6-4ffe-89d4-668f90243806","Type":"ContainerStarted","Data":"4bd99f104253431eab763d382262af8a238cef501d35fbf6f07d84480c320359"}
Nov 03 07:59:03.158775 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:03.158742    4470 generic.go:356] "Generic (PLEG): container finished" podID="b0f347c3-e6a6-4ffe-89d4-668f90243806" containerID="c15f71355129d2db3da602b469c9f8a2a7fed1c72b1a1ca349768145fa64478c" exitCode=0
Nov 03 07:59:03.159190 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:03.158807    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9" event={"ID":"b0f347c3-e6a6-4ffe-89d4-668f90243806","Type":"ContainerDied","Data":"c15f71355129d2db3da602b469c9f8a2a7fed1c72b1a1ca349768145fa64478c"}
Nov 03 07:59:04.301837 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:04.301813    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:04.391192 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:04.391160    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-xks-ca\") pod \"b0f347c3-e6a6-4ffe-89d4-668f90243806\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") "
Nov 03 07:59:04.391369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:04.391322    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-openshift-ca\") pod \"b0f347c3-e6a6-4ffe-89d4-668f90243806\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") "
Nov 03 07:59:04.391369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:04.391351    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f347c3-e6a6-4ffe-89d4-668f90243806-kube-api-access-r5tlw\" (UniqueName: \"kubernetes.io/projected/b0f347c3-e6a6-4ffe-89d4-668f90243806-kube-api-access-r5tlw\") pod \"b0f347c3-e6a6-4ffe-89d4-668f90243806\" (UID: \"b0f347c3-e6a6-4ffe-89d4-668f90243806\") "
Nov 03 07:59:04.391756 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:04.391713    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-openshift-ca" pod "b0f347c3-e6a6-4ffe-89d4-668f90243806" (UID: "b0f347c3-e6a6-4ffe-89d4-668f90243806"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:59:04.393332 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:04.393312    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-xks-ca" pod "b0f347c3-e6a6-4ffe-89d4-668f90243806" (UID: "b0f347c3-e6a6-4ffe-89d4-668f90243806"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:59:04.393498 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:04.393478    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f347c3-e6a6-4ffe-89d4-668f90243806-kube-api-access-r5tlw" pod "b0f347c3-e6a6-4ffe-89d4-668f90243806" (UID: "b0f347c3-e6a6-4ffe-89d4-668f90243806"). InnerVolumeSpecName "kube-api-access-r5tlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:59:04.492469 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:04.492386    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:59:04.492469 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:04.492414    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f347c3-e6a6-4ffe-89d4-668f90243806-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:59:04.492469 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:04.492425    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5tlw\" (UniqueName: \"kubernetes.io/projected/b0f347c3-e6a6-4ffe-89d4-668f90243806-kube-api-access-r5tlw\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:59:04.759198 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:59:04.759102    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 07:59:05.055900 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:05.055782    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"]
Nov 03 07:59:05.058603 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:05.058582    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d9v4z"]
Nov 03 07:59:05.171310 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:05.171275    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"
Nov 03 07:59:05.171485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:05.171312    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9" event={"ID":"b0f347c3-e6a6-4ffe-89d4-668f90243806","Type":"ContainerDied","Data":"4bd99f104253431eab763d382262af8a238cef501d35fbf6f07d84480c320359"}
Nov 03 07:59:05.171485 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:05.171356    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bd99f104253431eab763d382262af8a238cef501d35fbf6f07d84480c320359"
Nov 03 07:59:05.807508 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:05.807465    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994e457f-8114-4391-b93c-3ce35edc5543" path="/var/lib/kubelet/pods/994e457f-8114-4391-b93c-3ce35edc5543/volumes"
Nov 03 07:59:06.578023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:06.577983    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d" podUID="14a629b9-0bbc-4906-a654-747ac6e5eae1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:59:06.578023 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:06.578000    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr" podUID="6cfde453-d033-4e52-89b1-937cbd991e3c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 07:59:12.474863 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.474793    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:59:12.570079 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.570058    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:59:12.575120 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.575103    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cfde453-d033-4e52-89b1-937cbd991e3c-kube-api-access-lr6zt\" (UniqueName: \"kubernetes.io/projected/6cfde453-d033-4e52-89b1-937cbd991e3c-kube-api-access-lr6zt\") pod \"6cfde453-d033-4e52-89b1-937cbd991e3c\" (UID: \"6cfde453-d033-4e52-89b1-937cbd991e3c\") "
Nov 03 07:59:12.575252 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.575235    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cfde453-d033-4e52-89b1-937cbd991e3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cfde453-d033-4e52-89b1-937cbd991e3c-plugins-config-volume\") pod \"6cfde453-d033-4e52-89b1-937cbd991e3c\" (UID: \"6cfde453-d033-4e52-89b1-937cbd991e3c\") "
Nov 03 07:59:12.575644 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.575620    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cfde453-d033-4e52-89b1-937cbd991e3c-plugins-config-volume" pod "6cfde453-d033-4e52-89b1-937cbd991e3c" (UID: "6cfde453-d033-4e52-89b1-937cbd991e3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:59:12.575728 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.575714    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cfde453-d033-4e52-89b1-937cbd991e3c-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:59:12.577174 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.577152    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfde453-d033-4e52-89b1-937cbd991e3c-kube-api-access-lr6zt" pod "6cfde453-d033-4e52-89b1-937cbd991e3c" (UID: "6cfde453-d033-4e52-89b1-937cbd991e3c"). InnerVolumeSpecName "kube-api-access-lr6zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:59:12.676522 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.676487    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a629b9-0bbc-4906-a654-747ac6e5eae1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a629b9-0bbc-4906-a654-747ac6e5eae1-plugins-config-volume\") pod \"14a629b9-0bbc-4906-a654-747ac6e5eae1\" (UID: \"14a629b9-0bbc-4906-a654-747ac6e5eae1\") "
Nov 03 07:59:12.676703 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.676560    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a629b9-0bbc-4906-a654-747ac6e5eae1-kube-api-access-qpctj\" (UniqueName: \"kubernetes.io/projected/14a629b9-0bbc-4906-a654-747ac6e5eae1-kube-api-access-qpctj\") pod \"14a629b9-0bbc-4906-a654-747ac6e5eae1\" (UID: \"14a629b9-0bbc-4906-a654-747ac6e5eae1\") "
Nov 03 07:59:12.676879 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.676858    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr6zt\" (UniqueName: \"kubernetes.io/projected/6cfde453-d033-4e52-89b1-937cbd991e3c-kube-api-access-lr6zt\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:59:12.676919 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.676893    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a629b9-0bbc-4906-a654-747ac6e5eae1-plugins-config-volume" pod "14a629b9-0bbc-4906-a654-747ac6e5eae1" (UID: "14a629b9-0bbc-4906-a654-747ac6e5eae1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:59:12.678751 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.678725    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a629b9-0bbc-4906-a654-747ac6e5eae1-kube-api-access-qpctj" pod "14a629b9-0bbc-4906-a654-747ac6e5eae1" (UID: "14a629b9-0bbc-4906-a654-747ac6e5eae1"). InnerVolumeSpecName "kube-api-access-qpctj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:59:12.778586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.778501    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a629b9-0bbc-4906-a654-747ac6e5eae1-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:59:12.778586 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:12.778532    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpctj\" (UniqueName: \"kubernetes.io/projected/14a629b9-0bbc-4906-a654-747ac6e5eae1-kube-api-access-qpctj\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 07:59:13.215160 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.215110    4470 generic.go:356] "Generic (PLEG): container finished" podID="14a629b9-0bbc-4906-a654-747ac6e5eae1" containerID="13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6" exitCode=0
Nov 03 07:59:13.215339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.215186    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"
Nov 03 07:59:13.215339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.215221    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d" event={"ID":"14a629b9-0bbc-4906-a654-747ac6e5eae1","Type":"ContainerDied","Data":"13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6"}
Nov 03 07:59:13.215339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.215284    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d" event={"ID":"14a629b9-0bbc-4906-a654-747ac6e5eae1","Type":"ContainerDied","Data":"0800102f702fd6e1be44d839d99af4fd88dbea30c46a729af5ab4b4e9f2abc38"}
Nov 03 07:59:13.215339 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.215304    4470 scope.go:122] "RemoveContainer" containerID="13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6"
Nov 03 07:59:13.217010 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.216982    4470 generic.go:356] "Generic (PLEG): container finished" podID="6cfde453-d033-4e52-89b1-937cbd991e3c" containerID="3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df" exitCode=0
Nov 03 07:59:13.217254 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.217044    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"
Nov 03 07:59:13.217369 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.217068    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr" event={"ID":"6cfde453-d033-4e52-89b1-937cbd991e3c","Type":"ContainerDied","Data":"3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df"}
Nov 03 07:59:13.217433 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.217376    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr" event={"ID":"6cfde453-d033-4e52-89b1-937cbd991e3c","Type":"ContainerDied","Data":"091b963f9cf82a85daed3a99b94787a2c171653c717047b795935124120aa5cd"}
Nov 03 07:59:13.227616 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.227590    4470 scope.go:122] "RemoveContainer" containerID="13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6"
Nov 03 07:59:13.227871 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:59:13.227851    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6\": container with ID starting with 13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6 not found: ID does not exist" containerID="13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6"
Nov 03 07:59:13.227921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.227880    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6"} err="failed to get container status \"13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6\": rpc error: code = NotFound desc = could not find container \"13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6\": container with ID starting with 13c4d85de0165410bf064892355edc74d04842976d9360d0dbe4074abee2c7c6 not found: ID does not exist"
Nov 03 07:59:13.227921 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.227898    4470 scope.go:122] "RemoveContainer" containerID="3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df"
Nov 03 07:59:13.236153 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.236114    4470 scope.go:122] "RemoveContainer" containerID="3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df"
Nov 03 07:59:13.236435 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:59:13.236415    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df\": container with ID starting with 3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df not found: ID does not exist" containerID="3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df"
Nov 03 07:59:13.236487 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.236445    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df"} err="failed to get container status \"3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df\": rpc error: code = NotFound desc = could not find container \"3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df\": container with ID starting with 3b7cc6bd4700d9ee91079e892184ca81a650d4fe30971e08613b94a52c3fd8df not found: ID does not exist"
Nov 03 07:59:13.385733 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.385701    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"]
Nov 03 07:59:13.436308 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.436278    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr"]
Nov 03 07:59:13.503180 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.503085    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"]
Nov 03 07:59:13.508942 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.508912    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d"]
Nov 03 07:59:13.805656 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.805580    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a629b9-0bbc-4906-a654-747ac6e5eae1" path="/var/lib/kubelet/pods/14a629b9-0bbc-4906-a654-747ac6e5eae1/volumes"
Nov 03 07:59:13.805951 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:13.805938    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cfde453-d033-4e52-89b1-937cbd991e3c" path="/var/lib/kubelet/pods/6cfde453-d033-4e52-89b1-937cbd991e3c/volumes"
Nov 03 07:59:15.278843 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:15.278804    4470 scope.go:122] "RemoveContainer" containerID="4a301be39b0d514a460ca045dbad50e6ccb91b8f5051d25d1a67f56e2c09d512"
Nov 03 07:59:15.287501 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:15.287481    4470 scope.go:122] "RemoveContainer" containerID="df4295737be585f244f8f3da6a78bac1dabdd20ecef9eb78060998f5b251ce7e"
Nov 03 07:59:15.295352 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:15.295333    4470 scope.go:122] "RemoveContainer" containerID="ec1f1d6f7f772bc19e02427f212f3c499a7031c908fe96a71fcf6464fa038a41"
Nov 03 07:59:29.057347 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:29.057309    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"]
Nov 03 07:59:29.107744 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:29.107714    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nbrz8"]
Nov 03 07:59:29.805363 ip-10-0-129-24 kubenswrapper[4470]: I1103 07:59:29.805331    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe3eb44-fab8-4d9e-809d-4511e2f48120" path="/var/lib/kubelet/pods/cbe3eb44-fab8-4d9e-809d-4511e2f48120/volumes"
Nov 03 07:59:34.759321 ip-10-0-129-24 kubenswrapper[4470]: E1103 07:59:34.759286    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:00:00.173151 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.173090    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"]
Nov 03 08:00:00.173654 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.173556    4470 cpu_manager.go:414] "containerMap: removing container" podUID="6cfde453-d033-4e52-89b1-937cbd991e3c" containerName="payload-processing"
Nov 03 08:00:00.173654 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.173573    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cfde453-d033-4e52-89b1-937cbd991e3c" containerName="payload-processing"
Nov 03 08:00:00.173654 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.173595    4470 cpu_manager.go:414] "containerMap: removing container" podUID="14a629b9-0bbc-4906-a654-747ac6e5eae1" containerName="payload-pre-processing"
Nov 03 08:00:00.173654 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.173600    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a629b9-0bbc-4906-a654-747ac6e5eae1" containerName="payload-pre-processing"
Nov 03 08:00:00.173654 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.173613    4470 cpu_manager.go:414] "containerMap: removing container" podUID="b0f347c3-e6a6-4ffe-89d4-668f90243806" containerName="revoke-keys"
Nov 03 08:00:00.173654 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.173618    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f347c3-e6a6-4ffe-89d4-668f90243806" containerName="revoke-keys"
Nov 03 08:00:00.173864 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.173681    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cfde453-d033-4e52-89b1-937cbd991e3c" containerName="payload-processing"
Nov 03 08:00:00.173864 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.173693    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a629b9-0bbc-4906-a654-747ac6e5eae1" containerName="payload-pre-processing"
Nov 03 08:00:00.173864 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.173700    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f347c3-e6a6-4ffe-89d4-668f90243806" containerName="revoke-keys"
Nov 03 08:00:00.183658 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.183625    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"
Nov 03 08:00:00.224770 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.224726    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgh8w\" (UniqueName: \"kubernetes.io/projected/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-kube-api-access-xgh8w\") pod \"maas-api-key-cleanup-29894880-6r4lx\" (UID: \"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"
Nov 03 08:00:00.224972 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.224902    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894880-6r4lx\" (UID: \"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"
Nov 03 08:00:00.235586 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.235552    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"]
Nov 03 08:00:00.283697 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.283661    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:00:00.283967 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.283877    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:00:00.283967 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.283884    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 08:00:00.326206 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.326167    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgh8w\" (UniqueName: \"kubernetes.io/projected/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-kube-api-access-xgh8w\") pod \"maas-api-key-cleanup-29894880-6r4lx\" (UID: \"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"
Nov 03 08:00:00.326396 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.326290    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894880-6r4lx\" (UID: \"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"
Nov 03 08:00:00.326882 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.326864    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894880-6r4lx\" (UID: \"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"
Nov 03 08:00:00.392054 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.392022    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgh8w\" (UniqueName: \"kubernetes.io/projected/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-kube-api-access-xgh8w\") pod \"maas-api-key-cleanup-29894880-6r4lx\" (UID: \"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"
Nov 03 08:00:00.497317 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.497225    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"
Nov 03 08:00:00.874295 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:00.874260    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"]
Nov 03 08:00:00.875730 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:00:00.875701    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc7e0e2_98c0_48e1_99e2_0dc041b2b59d.slice/crio-2b3e338545e9e96155a2fa203391357fb09bbc549456ba488f0ee9eac0c81fcc WatchSource:0}: Error finding container 2b3e338545e9e96155a2fa203391357fb09bbc549456ba488f0ee9eac0c81fcc: Status 404 returned error can't find the container with id 2b3e338545e9e96155a2fa203391357fb09bbc549456ba488f0ee9eac0c81fcc
Nov 03 08:00:01.499020 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:01.498934    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx" event={"ID":"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d","Type":"ContainerStarted","Data":"218a9860db747613087f84255d9dff627e7b480fd977acd08bb59847b89d3e36"}
Nov 03 08:00:01.499020 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:01.498992    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx" event={"ID":"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d","Type":"ContainerStarted","Data":"2b3e338545e9e96155a2fa203391357fb09bbc549456ba488f0ee9eac0c81fcc"}
Nov 03 08:00:01.618968 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:01.618914    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx" podStartSLOduration=1.61890099 podStartE2EDuration="1.61890099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:00:01.618669063 +0000 UTC m=+3653.951038060" watchObservedRunningTime="2026-11-03 08:00:01.61890099 +0000 UTC m=+3653.951270005"
Nov 03 08:00:02.506523 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:02.506485    4470 generic.go:356] "Generic (PLEG): container finished" podID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerID="218a9860db747613087f84255d9dff627e7b480fd977acd08bb59847b89d3e36" exitCode=60
Nov 03 08:00:02.507018 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:02.506546    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx" event={"ID":"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d","Type":"ContainerDied","Data":"218a9860db747613087f84255d9dff627e7b480fd977acd08bb59847b89d3e36"}
Nov 03 08:00:02.507018 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:02.506864    4470 scope.go:122] "RemoveContainer" containerID="218a9860db747613087f84255d9dff627e7b480fd977acd08bb59847b89d3e36"
Nov 03 08:00:03.513198 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:03.513085    4470 generic.go:356] "Generic (PLEG): container finished" podID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerID="82031c9ab4b1a916f3583b1abc1aab33d88e551dbe2800f0507041ef91c98569" exitCode=60
Nov 03 08:00:03.513198 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:03.513125    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx" event={"ID":"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d","Type":"ContainerDied","Data":"82031c9ab4b1a916f3583b1abc1aab33d88e551dbe2800f0507041ef91c98569"}
Nov 03 08:00:03.513198 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:03.513192    4470 scope.go:122] "RemoveContainer" containerID="218a9860db747613087f84255d9dff627e7b480fd977acd08bb59847b89d3e36"
Nov 03 08:00:03.513779 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:03.513466    4470 scope.go:122] "RemoveContainer" containerID="82031c9ab4b1a916f3583b1abc1aab33d88e551dbe2800f0507041ef91c98569"
Nov 03 08:00:03.513779 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:00:03.513741    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894880-6r4lx_odh-ai-gateway-infra(8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d"
Nov 03 08:00:04.520427 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:04.520395    4470 scope.go:122] "RemoveContainer" containerID="82031c9ab4b1a916f3583b1abc1aab33d88e551dbe2800f0507041ef91c98569"
Nov 03 08:00:04.520998 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:00:04.520721    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894880-6r4lx_odh-ai-gateway-infra(8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d"
Nov 03 08:00:04.760145 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:00:04.760094    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:00:15.332316 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:15.332284    4470 scope.go:122] "RemoveContainer" containerID="bf02ee7b8e576326f5f821aeb56f016d5ab4b25c8932e1b15e961e7b71d6ab00"
Nov 03 08:00:15.341115 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:15.341093    4470 scope.go:122] "RemoveContainer" containerID="5cf81e68862e8da02cf33669cb075375d77e44e4bc7695454e0da46588e482c6"
Nov 03 08:00:16.799190 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:16.799157    4470 scope.go:122] "RemoveContainer" containerID="82031c9ab4b1a916f3583b1abc1aab33d88e551dbe2800f0507041ef91c98569"
Nov 03 08:00:17.595334 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:17.595299    4470 generic.go:356] "Generic (PLEG): container finished" podID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerID="7ccdedfa5d80fb25c3fce04f3e172318cc4465cbfa51b78199edd48ed9a7211a" exitCode=60
Nov 03 08:00:17.595534 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:17.595336    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx" event={"ID":"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d","Type":"ContainerDied","Data":"7ccdedfa5d80fb25c3fce04f3e172318cc4465cbfa51b78199edd48ed9a7211a"}
Nov 03 08:00:17.595534 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:17.595407    4470 scope.go:122] "RemoveContainer" containerID="82031c9ab4b1a916f3583b1abc1aab33d88e551dbe2800f0507041ef91c98569"
Nov 03 08:00:17.595755 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:17.595742    4470 scope.go:122] "RemoveContainer" containerID="7ccdedfa5d80fb25c3fce04f3e172318cc4465cbfa51b78199edd48ed9a7211a"
Nov 03 08:00:17.595995 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:00:17.595978    4470 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894880-6r4lx_odh-ai-gateway-infra(8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d"
Nov 03 08:00:18.727480 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:18.727445    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"]
Nov 03 08:00:18.865675 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:18.865652    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"
Nov 03 08:00:19.007184 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:19.007075    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-kube-api-access-xgh8w\" (UniqueName: \"kubernetes.io/projected/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-kube-api-access-xgh8w\") pod \"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d\" (UID: \"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d\") "
Nov 03 08:00:19.007184 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:19.007116    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-maas-api-service-ca\") pod \"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d\" (UID: \"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d\") "
Nov 03 08:00:19.007621 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:19.007585    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-maas-api-service-ca" pod "8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" (UID: "8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:00:19.009467 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:19.009439    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-kube-api-access-xgh8w" pod "8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" (UID: "8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d"). InnerVolumeSpecName "kube-api-access-xgh8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:00:19.108088 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:19.108050    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgh8w\" (UniqueName: \"kubernetes.io/projected/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-kube-api-access-xgh8w\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:00:19.108088 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:19.108082    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d-maas-api-service-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:00:19.608339 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:19.608298    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx" event={"ID":"8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d","Type":"ContainerDied","Data":"2b3e338545e9e96155a2fa203391357fb09bbc549456ba488f0ee9eac0c81fcc"}
Nov 03 08:00:19.608339 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:19.608330    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"
Nov 03 08:00:19.608578 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:19.608373    4470 scope.go:122] "RemoveContainer" containerID="7ccdedfa5d80fb25c3fce04f3e172318cc4465cbfa51b78199edd48ed9a7211a"
Nov 03 08:00:19.780155 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:19.780102    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"]
Nov 03 08:00:19.831033 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:19.831002    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894880-6r4lx"]
Nov 03 08:00:21.805428 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:00:21.805397    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" path="/var/lib/kubelet/pods/8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d/volumes"
Nov 03 08:00:34.758803 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:00:34.758759    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:01:04.741552 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.741519    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"]
Nov 03 08:01:04.741986 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.741943    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerName="cleanup"
Nov 03 08:01:04.741986 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.741955    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerName="cleanup"
Nov 03 08:01:04.741986 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.741966    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerName="cleanup"
Nov 03 08:01:04.741986 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.741970    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerName="cleanup"
Nov 03 08:01:04.741986 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.741981    4470 cpu_manager.go:414] "containerMap: removing container" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerName="cleanup"
Nov 03 08:01:04.741986 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.741985    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerName="cleanup"
Nov 03 08:01:04.742222 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.742073    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerName="cleanup"
Nov 03 08:01:04.742222 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.742079    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerName="cleanup"
Nov 03 08:01:04.745366 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.745350    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:04.751343 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.751316    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cfmhw\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:04.751343 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.751355    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cfmhw\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:04.751593 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.751512    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9r5\" (UniqueName: \"kubernetes.io/projected/ea87ef0d-bc6b-4046-b16e-e5eb953be964-kube-api-access-nw9r5\") pod \"maas-api-revoke-sub-d576e68b-cfmhw\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:04.761480 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:01:04.761450    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:01:04.849707 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.849669    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"]
Nov 03 08:01:04.852233 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.852214    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9r5\" (UniqueName: \"kubernetes.io/projected/ea87ef0d-bc6b-4046-b16e-e5eb953be964-kube-api-access-nw9r5\") pod \"maas-api-revoke-sub-d576e68b-cfmhw\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:04.852358 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.852284    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cfmhw\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:04.852358 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.852312    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cfmhw\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:04.854108 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.854081    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 08:01:04.854234 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.854101    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:01:04.854234 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.854114    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:01:04.854234 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.854106    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:01:04.864112 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.864089    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cfmhw\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:04.865948 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.865918    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cfmhw\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:04.916589 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:04.916558    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9r5\" (UniqueName: \"kubernetes.io/projected/ea87ef0d-bc6b-4046-b16e-e5eb953be964-kube-api-access-nw9r5\") pod \"maas-api-revoke-sub-d576e68b-cfmhw\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:05.060310 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:05.060236    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:05.434560 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:05.434534    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"]
Nov 03 08:01:05.876075 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:05.876042    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw" event={"ID":"ea87ef0d-bc6b-4046-b16e-e5eb953be964","Type":"ContainerStarted","Data":"2a60562cda192dfa388516438106f65aca0f4bcd7c3d3edde2c43511bfb26c35"}
Nov 03 08:01:05.876488 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:05.876092    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw" event={"ID":"ea87ef0d-bc6b-4046-b16e-e5eb953be964","Type":"ContainerStarted","Data":"3977746b4cacf71252f70cbe767d5a71a25a69269c4593ba6d36cf1480156813"}
Nov 03 08:01:05.996693 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:05.996637    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw" podStartSLOduration=1.996623036 podStartE2EDuration="1.996623036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:05.995635655 +0000 UTC m=+3718.328004656" watchObservedRunningTime="2026-11-03 08:01:05.996623036 +0000 UTC m=+3718.328992029"
Nov 03 08:01:06.882620 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:06.882587    4470 generic.go:356] "Generic (PLEG): container finished" podID="ea87ef0d-bc6b-4046-b16e-e5eb953be964" containerID="2a60562cda192dfa388516438106f65aca0f4bcd7c3d3edde2c43511bfb26c35" exitCode=0
Nov 03 08:01:06.883041 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:06.882678    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw" event={"ID":"ea87ef0d-bc6b-4046-b16e-e5eb953be964","Type":"ContainerDied","Data":"2a60562cda192dfa388516438106f65aca0f4bcd7c3d3edde2c43511bfb26c35"}
Nov 03 08:01:08.033453 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.033425    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:08.056902 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.056863    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"]
Nov 03 08:01:08.066459 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.066425    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5pskn"]
Nov 03 08:01:08.076302 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.076280    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea87ef0d-bc6b-4046-b16e-e5eb953be964-kube-api-access-nw9r5\" (UniqueName: \"kubernetes.io/projected/ea87ef0d-bc6b-4046-b16e-e5eb953be964-kube-api-access-nw9r5\") pod \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") "
Nov 03 08:01:08.076404 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.076314    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-xks-ca\") pod \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") "
Nov 03 08:01:08.076404 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.076346    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-openshift-ca\") pod \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\" (UID: \"ea87ef0d-bc6b-4046-b16e-e5eb953be964\") "
Nov 03 08:01:08.076708 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.076685    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-openshift-ca" pod "ea87ef0d-bc6b-4046-b16e-e5eb953be964" (UID: "ea87ef0d-bc6b-4046-b16e-e5eb953be964"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:01:08.078539 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.078516    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea87ef0d-bc6b-4046-b16e-e5eb953be964-kube-api-access-nw9r5" pod "ea87ef0d-bc6b-4046-b16e-e5eb953be964" (UID: "ea87ef0d-bc6b-4046-b16e-e5eb953be964"). InnerVolumeSpecName "kube-api-access-nw9r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:01:08.078646 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.078537    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-xks-ca" pod "ea87ef0d-bc6b-4046-b16e-e5eb953be964" (UID: "ea87ef0d-bc6b-4046-b16e-e5eb953be964"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:01:08.177615 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.177535    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9r5\" (UniqueName: \"kubernetes.io/projected/ea87ef0d-bc6b-4046-b16e-e5eb953be964-kube-api-access-nw9r5\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:01:08.177615 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.177564    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:01:08.177615 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.177574    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea87ef0d-bc6b-4046-b16e-e5eb953be964-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:01:08.894897 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.894864    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"
Nov 03 08:01:08.894897 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.894884    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw" event={"ID":"ea87ef0d-bc6b-4046-b16e-e5eb953be964","Type":"ContainerDied","Data":"3977746b4cacf71252f70cbe767d5a71a25a69269c4593ba6d36cf1480156813"}
Nov 03 08:01:08.895197 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:08.894928    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3977746b4cacf71252f70cbe767d5a71a25a69269c4593ba6d36cf1480156813"
Nov 03 08:01:09.806448 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:09.806416    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21540773-a8f7-4bab-9bd5-570a6a90fa78" path="/var/lib/kubelet/pods/21540773-a8f7-4bab-9bd5-570a6a90fa78/volumes"
Nov 03 08:01:15.364760 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:15.364727    4470 scope.go:122] "RemoveContainer" containerID="ccb248af23ba27530e782272169269f42d4e3e265385409ec2d834dc1502a3ba"
Nov 03 08:01:15.373434 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:15.373409    4470 scope.go:122] "RemoveContainer" containerID="5cac1b65a0c450256e66841432dae400b30ca26f538a9dac727d8b3c1882a371"
Nov 03 08:01:15.381439 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:15.381422    4470 scope.go:122] "RemoveContainer" containerID="f1302c4a44776345521380abe65588dbb6464e4b119d26ff37f8d4eb888615d9"
Nov 03 08:01:19.063305 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:19.063226    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"]
Nov 03 08:01:19.113770 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:19.113738    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7vp9z"]
Nov 03 08:01:19.805446 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:19.805413    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73d0b64-478d-41a5-9256-88dfa0879aa7" path="/var/lib/kubelet/pods/c73d0b64-478d-41a5-9256-88dfa0879aa7/volumes"
Nov 03 08:01:24.760170 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:24.759966    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:01:24.760170 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:24.760030    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" podUID="4dfa6c32-f24d-4d1f-96ff-df222dec2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:01:33.744332 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.744296    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"]
Nov 03 08:01:33.746725 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.744726    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ea87ef0d-bc6b-4046-b16e-e5eb953be964" containerName="revoke-keys"
Nov 03 08:01:33.746725 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.744736    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea87ef0d-bc6b-4046-b16e-e5eb953be964" containerName="revoke-keys"
Nov 03 08:01:33.746725 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.744806    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc7e0e2-98c0-48e1-99e2-0dc041b2b59d" containerName="cleanup"
Nov 03 08:01:33.746725 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.744816    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea87ef0d-bc6b-4046-b16e-e5eb953be964" containerName="revoke-keys"
Nov 03 08:01:33.747723 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.747708    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:33.806921 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.806888    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"]
Nov 03 08:01:33.855368 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.855333    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:01:33.855368 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.855358    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:01:33.855368 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.855338    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:01:33.855620 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.855388    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 08:01:33.876282 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.876254    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngssg\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:33.876431 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.876303    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngssg\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:33.876475 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.876441    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5f5n\" (UniqueName: \"kubernetes.io/projected/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-kube-api-access-w5f5n\") pod \"maas-api-revoke-sub-d2bc93bc-ngssg\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:33.977897 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.977863    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngssg\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:33.978080 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.977905    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngssg\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:33.978080 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.977956    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5f5n\" (UniqueName: \"kubernetes.io/projected/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-kube-api-access-w5f5n\") pod \"maas-api-revoke-sub-d2bc93bc-ngssg\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:33.978781 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.978715    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngssg\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:33.980560 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:33.980532    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ngssg\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:34.042691 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:34.042618    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5f5n\" (UniqueName: \"kubernetes.io/projected/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-kube-api-access-w5f5n\") pod \"maas-api-revoke-sub-d2bc93bc-ngssg\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:34.060306 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:34.060280    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:34.431951 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:34.431910    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"]
Nov 03 08:01:34.471954 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:01:34.471923    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7b6cda_cf4e_4cbb_90a3_cfd6c92b204c.slice/crio-2f589f2b4c8908722c5bf1deb29530180fb9b7361334f2fd324dc90585f9becd WatchSource:0}: Error finding container 2f589f2b4c8908722c5bf1deb29530180fb9b7361334f2fd324dc90585f9becd: Status 404 returned error can't find the container with id 2f589f2b4c8908722c5bf1deb29530180fb9b7361334f2fd324dc90585f9becd
Nov 03 08:01:34.761277 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:01:34.761242    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:01:35.044583 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:35.044495    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg" event={"ID":"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c","Type":"ContainerStarted","Data":"76ccd1745ebdc624452c25327f07f1b70fe4297943b0ddd40ac829d3baa8ba33"}
Nov 03 08:01:35.044583 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:35.044553    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg" event={"ID":"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c","Type":"ContainerStarted","Data":"2f589f2b4c8908722c5bf1deb29530180fb9b7361334f2fd324dc90585f9becd"}
Nov 03 08:01:35.166280 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:35.166229    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg" podStartSLOduration=2.166215093 podStartE2EDuration="2.166215093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:35.164734838 +0000 UTC m=+3747.497103833" watchObservedRunningTime="2026-11-03 08:01:35.166215093 +0000 UTC m=+3747.498584077"
Nov 03 08:01:36.050552 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:36.050522    4470 generic.go:356] "Generic (PLEG): container finished" podID="fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c" containerID="76ccd1745ebdc624452c25327f07f1b70fe4297943b0ddd40ac829d3baa8ba33" exitCode=0
Nov 03 08:01:36.050951 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:36.050589    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg" event={"ID":"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c","Type":"ContainerDied","Data":"76ccd1745ebdc624452c25327f07f1b70fe4297943b0ddd40ac829d3baa8ba33"}
Nov 03 08:01:37.191856 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:37.191833    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:37.311881 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:37.311800    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-kube-api-access-w5f5n\" (UniqueName: \"kubernetes.io/projected/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-kube-api-access-w5f5n\") pod \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") "
Nov 03 08:01:37.312063 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:37.311890    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-openshift-ca\") pod \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") "
Nov 03 08:01:37.312063 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:37.311979    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-xks-ca\") pod \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\" (UID: \"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c\") "
Nov 03 08:01:37.312331 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:37.312308    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-openshift-ca" pod "fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c" (UID: "fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:01:37.314531 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:37.314508    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-xks-ca" pod "fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c" (UID: "fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:01:37.314697 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:37.314671    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-kube-api-access-w5f5n" pod "fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c" (UID: "fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c"). InnerVolumeSpecName "kube-api-access-w5f5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:01:37.413000 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:37.412968    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:01:37.413000 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:37.412997    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5f5n\" (UniqueName: \"kubernetes.io/projected/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-kube-api-access-w5f5n\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:01:37.413236 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:37.413009    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:01:38.061600 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:38.061563    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg" event={"ID":"fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c","Type":"ContainerDied","Data":"2f589f2b4c8908722c5bf1deb29530180fb9b7361334f2fd324dc90585f9becd"}
Nov 03 08:01:38.061600 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:38.061588    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"
Nov 03 08:01:38.061834 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:38.061622    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f589f2b4c8908722c5bf1deb29530180fb9b7361334f2fd324dc90585f9becd"
Nov 03 08:01:54.570066 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.570032    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"]
Nov 03 08:01:54.570497 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.570434    4470 cpu_manager.go:414] "containerMap: removing container" podUID="fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c" containerName="revoke-keys"
Nov 03 08:01:54.570497 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.570445    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c" containerName="revoke-keys"
Nov 03 08:01:54.570583 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.570535    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c" containerName="revoke-keys"
Nov 03 08:01:54.573711 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.573695    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:01:54.632234 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.632202    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"]
Nov 03 08:01:54.680091 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.680060    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b435d8\""
Nov 03 08:01:54.680291 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.680125    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-6929e6fb-dockercfg-655v6\""
Nov 03 08:01:54.688596 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.688565    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz429\" (UniqueName: \"kubernetes.io/projected/3fb568ca-b9e1-4037-83af-998712ef7ca1-kube-api-access-zz429\") pod \"payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw\" (UID: \"3fb568ca-b9e1-4037-83af-998712ef7ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:01:54.688754 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.688627    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb568ca-b9e1-4037-83af-998712ef7ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw\" (UID: \"3fb568ca-b9e1-4037-83af-998712ef7ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:01:54.733186 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.733119    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"]
Nov 03 08:01:54.736707 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.736691    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:01:54.766111 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.766081    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"]
Nov 03 08:01:54.789382 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.789344    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6462404-8ce7-4ce9-abab-5471fb3f0275-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm\" (UID: \"f6462404-8ce7-4ce9-abab-5471fb3f0275\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:01:54.789563 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.789398    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zth\" (UniqueName: \"kubernetes.io/projected/f6462404-8ce7-4ce9-abab-5471fb3f0275-kube-api-access-m9zth\") pod \"payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm\" (UID: \"f6462404-8ce7-4ce9-abab-5471fb3f0275\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:01:54.789563 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.789546    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz429\" (UniqueName: \"kubernetes.io/projected/3fb568ca-b9e1-4037-83af-998712ef7ca1-kube-api-access-zz429\") pod \"payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw\" (UID: \"3fb568ca-b9e1-4037-83af-998712ef7ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:01:54.789692 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.789635    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb568ca-b9e1-4037-83af-998712ef7ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw\" (UID: \"3fb568ca-b9e1-4037-83af-998712ef7ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:01:54.790312 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.790290    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb568ca-b9e1-4037-83af-998712ef7ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw\" (UID: \"3fb568ca-b9e1-4037-83af-998712ef7ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:01:54.853914 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.853841    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz429\" (UniqueName: \"kubernetes.io/projected/3fb568ca-b9e1-4037-83af-998712ef7ca1-kube-api-access-zz429\") pod \"payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw\" (UID: \"3fb568ca-b9e1-4037-83af-998712ef7ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:01:54.886690 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.886662    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:01:54.890108 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.890088    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6462404-8ce7-4ce9-abab-5471fb3f0275-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm\" (UID: \"f6462404-8ce7-4ce9-abab-5471fb3f0275\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:01:54.890188 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.890165    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zth\" (UniqueName: \"kubernetes.io/projected/f6462404-8ce7-4ce9-abab-5471fb3f0275-kube-api-access-m9zth\") pod \"payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm\" (UID: \"f6462404-8ce7-4ce9-abab-5471fb3f0275\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:01:54.890731 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.890711    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6462404-8ce7-4ce9-abab-5471fb3f0275-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm\" (UID: \"f6462404-8ce7-4ce9-abab-5471fb3f0275\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:01:54.956055 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:54.956028    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zth\" (UniqueName: \"kubernetes.io/projected/f6462404-8ce7-4ce9-abab-5471fb3f0275-kube-api-access-m9zth\") pod \"payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm\" (UID: \"f6462404-8ce7-4ce9-abab-5471fb3f0275\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:01:55.049430 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:55.049385    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:01:55.260555 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:55.260528    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"]
Nov 03 08:01:55.261769 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:01:55.261747    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb568ca_b9e1_4037_83af_998712ef7ca1.slice/crio-13310badbb8c4421dbde922e3a306c305924be707dba398fe065e57be3aedc8c WatchSource:0}: Error finding container 13310badbb8c4421dbde922e3a306c305924be707dba398fe065e57be3aedc8c: Status 404 returned error can't find the container with id 13310badbb8c4421dbde922e3a306c305924be707dba398fe065e57be3aedc8c
Nov 03 08:01:55.422879 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:55.422852    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"]
Nov 03 08:01:55.424976 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:01:55.424948    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6462404_8ce7_4ce9_abab_5471fb3f0275.slice/crio-88c349eaaa9a84b17cee272f71d2fe7ae073694e55350f171b5ac5e54fdd81c7 WatchSource:0}: Error finding container 88c349eaaa9a84b17cee272f71d2fe7ae073694e55350f171b5ac5e54fdd81c7: Status 404 returned error can't find the container with id 88c349eaaa9a84b17cee272f71d2fe7ae073694e55350f171b5ac5e54fdd81c7
Nov 03 08:01:56.153605 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:56.153566    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm" event={"ID":"f6462404-8ce7-4ce9-abab-5471fb3f0275","Type":"ContainerStarted","Data":"8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3"}
Nov 03 08:01:56.154091 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:56.153623    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm" event={"ID":"f6462404-8ce7-4ce9-abab-5471fb3f0275","Type":"ContainerStarted","Data":"88c349eaaa9a84b17cee272f71d2fe7ae073694e55350f171b5ac5e54fdd81c7"}
Nov 03 08:01:56.154091 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:56.153752    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:01:56.155379 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:56.155358    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw" event={"ID":"3fb568ca-b9e1-4037-83af-998712ef7ca1","Type":"ContainerStarted","Data":"f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2"}
Nov 03 08:01:56.155454 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:56.155395    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw" event={"ID":"3fb568ca-b9e1-4037-83af-998712ef7ca1","Type":"ContainerStarted","Data":"13310badbb8c4421dbde922e3a306c305924be707dba398fe065e57be3aedc8c"}
Nov 03 08:01:56.155507 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:56.155488    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:01:56.274271 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:56.274220    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm" podStartSLOduration=2.2742032070000002 podStartE2EDuration="2.274203207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:56.273184836 +0000 UTC m=+3768.605553831" watchObservedRunningTime="2026-11-03 08:01:56.274203207 +0000 UTC m=+3768.606572281"
Nov 03 08:01:56.394881 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:01:56.394829    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw" podStartSLOduration=2.394810236 podStartE2EDuration="2.394810236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:56.392456877 +0000 UTC m=+3768.724825871" watchObservedRunningTime="2026-11-03 08:01:56.394810236 +0000 UTC m=+3768.727179231"
Nov 03 08:02:04.759296 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:02:04.759250    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:02:07.163205 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:02:07.163177    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:02:07.163738 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:02:07.163612    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:02:15.413786 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:02:15.413751    4470 scope.go:122] "RemoveContainer" containerID="4bb599a3ed86e7ec6e02bfa7d79b7686dd6fc5f31d58db45514c5528d6dd6f5e"
Nov 03 08:02:34.758823 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:02:34.758782    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:03:04.656319 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.656283    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"]
Nov 03 08:03:04.660058 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.660042    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:04.699702 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.699672    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwrl9\" (UniqueName: \"kubernetes.io/projected/5c931178-6eac-46b6-9c3e-308fe2e37e24-kube-api-access-vwrl9\") pod \"maas-api-revoke-sub-48198fc4-hlzcj\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:04.699892 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.699713    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48198fc4-hlzcj\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:04.699892 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.699824    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48198fc4-hlzcj\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:04.759788 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:03:04.759751    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:03:04.764217 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.764190    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"]
Nov 03 08:03:04.766660 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.766638    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:03:04.766782 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.766706    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:03:04.766782 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.766754    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:03:04.766902 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.766877    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 08:03:04.800484 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.800451    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwrl9\" (UniqueName: \"kubernetes.io/projected/5c931178-6eac-46b6-9c3e-308fe2e37e24-kube-api-access-vwrl9\") pod \"maas-api-revoke-sub-48198fc4-hlzcj\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:04.800641 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.800497    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48198fc4-hlzcj\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:04.800641 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.800547    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48198fc4-hlzcj\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:04.801269 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.801246    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48198fc4-hlzcj\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:04.802969 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.802949    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48198fc4-hlzcj\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:04.864913 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.864884    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwrl9\" (UniqueName: \"kubernetes.io/projected/5c931178-6eac-46b6-9c3e-308fe2e37e24-kube-api-access-vwrl9\") pod \"maas-api-revoke-sub-48198fc4-hlzcj\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:04.973158 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:04.973063    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:05.350062 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:05.350040    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"]
Nov 03 08:03:05.352739 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:03:05.352704    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c931178_6eac_46b6_9c3e_308fe2e37e24.slice/crio-3e4b443049b6ae89513ad46b588241b8f84983da9da4c095eeb60dacf9af0d70 WatchSource:0}: Error finding container 3e4b443049b6ae89513ad46b588241b8f84983da9da4c095eeb60dacf9af0d70: Status 404 returned error can't find the container with id 3e4b443049b6ae89513ad46b588241b8f84983da9da4c095eeb60dacf9af0d70
Nov 03 08:03:05.519846 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:05.519812    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj" event={"ID":"5c931178-6eac-46b6-9c3e-308fe2e37e24","Type":"ContainerStarted","Data":"3e4b443049b6ae89513ad46b588241b8f84983da9da4c095eeb60dacf9af0d70"}
Nov 03 08:03:06.526468 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:06.526390    4470 generic.go:356] "Generic (PLEG): container finished" podID="5c931178-6eac-46b6-9c3e-308fe2e37e24" containerID="5488a65d5fc99fc6979dda312e2bb17f075c6abffba12961f013ae2cb7a291ba" exitCode=0
Nov 03 08:03:06.526468 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:06.526437    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj" event={"ID":"5c931178-6eac-46b6-9c3e-308fe2e37e24","Type":"ContainerDied","Data":"5488a65d5fc99fc6979dda312e2bb17f075c6abffba12961f013ae2cb7a291ba"}
Nov 03 08:03:07.666871 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:07.666846    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:07.725387 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:07.725352    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c931178-6eac-46b6-9c3e-308fe2e37e24-kube-api-access-vwrl9\" (UniqueName: \"kubernetes.io/projected/5c931178-6eac-46b6-9c3e-308fe2e37e24-kube-api-access-vwrl9\") pod \"5c931178-6eac-46b6-9c3e-308fe2e37e24\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") "
Nov 03 08:03:07.725600 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:07.725400    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-xks-ca\") pod \"5c931178-6eac-46b6-9c3e-308fe2e37e24\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") "
Nov 03 08:03:07.725600 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:07.725461    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-openshift-ca\") pod \"5c931178-6eac-46b6-9c3e-308fe2e37e24\" (UID: \"5c931178-6eac-46b6-9c3e-308fe2e37e24\") "
Nov 03 08:03:07.725849 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:07.725826    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-openshift-ca" pod "5c931178-6eac-46b6-9c3e-308fe2e37e24" (UID: "5c931178-6eac-46b6-9c3e-308fe2e37e24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:03:07.727713 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:07.727690    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c931178-6eac-46b6-9c3e-308fe2e37e24-kube-api-access-vwrl9" pod "5c931178-6eac-46b6-9c3e-308fe2e37e24" (UID: "5c931178-6eac-46b6-9c3e-308fe2e37e24"). InnerVolumeSpecName "kube-api-access-vwrl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:03:07.727797 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:07.727736    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-xks-ca" pod "5c931178-6eac-46b6-9c3e-308fe2e37e24" (UID: "5c931178-6eac-46b6-9c3e-308fe2e37e24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:03:07.827104 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:07.827077    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:03:07.827104 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:07.827104    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwrl9\" (UniqueName: \"kubernetes.io/projected/5c931178-6eac-46b6-9c3e-308fe2e37e24-kube-api-access-vwrl9\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:03:07.827317 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:07.827118    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c931178-6eac-46b6-9c3e-308fe2e37e24-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:03:08.538182 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:08.538152    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"
Nov 03 08:03:08.538387 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:08.538177    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj" event={"ID":"5c931178-6eac-46b6-9c3e-308fe2e37e24","Type":"ContainerDied","Data":"3e4b443049b6ae89513ad46b588241b8f84983da9da4c095eeb60dacf9af0d70"}
Nov 03 08:03:08.538387 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:08.538266    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e4b443049b6ae89513ad46b588241b8f84983da9da4c095eeb60dacf9af0d70"
Nov 03 08:03:15.434685 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:15.434653    4470 scope.go:122] "RemoveContainer" containerID="a4ff4c859bc36cbcf2efb293d736dd95a37df412fcb7f705d5502caf3bf713c3"
Nov 03 08:03:15.443567 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:15.443549    4470 scope.go:122] "RemoveContainer" containerID="ef1c257d9d123cb3da9e1d8cd228e6d1b39f544486f3c4361903520eff00cd90"
Nov 03 08:03:16.392029 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.391995    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"]
Nov 03 08:03:16.392442 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.392430    4470 cpu_manager.go:414] "containerMap: removing container" podUID="5c931178-6eac-46b6-9c3e-308fe2e37e24" containerName="revoke-keys"
Nov 03 08:03:16.392495 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.392444    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c931178-6eac-46b6-9c3e-308fe2e37e24" containerName="revoke-keys"
Nov 03 08:03:16.392535 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.392525    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c931178-6eac-46b6-9c3e-308fe2e37e24" containerName="revoke-keys"
Nov 03 08:03:16.397027 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.397010    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:16.456579 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.456545    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"]
Nov 03 08:03:16.503803 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.503765    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:03:16.503956 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.503822    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:03:16.503956 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.503842    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:03:16.504217 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.504202    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 08:03:16.508888 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.508866    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5bc9\" (UniqueName: \"kubernetes.io/projected/15f1344d-5891-41db-9037-7e6f84a75530-kube-api-access-c5bc9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:16.508945 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.508932    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:16.509016 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.509000    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:16.610518 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.610480    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:16.610729 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.610522    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:16.610729 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.610672    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5bc9\" (UniqueName: \"kubernetes.io/projected/15f1344d-5891-41db-9037-7e6f84a75530-kube-api-access-c5bc9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:16.611107 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.611074    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:16.612979 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.612952    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:16.675465 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.675401    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5bc9\" (UniqueName: \"kubernetes.io/projected/15f1344d-5891-41db-9037-7e6f84a75530-kube-api-access-c5bc9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:16.709366 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:16.709334    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:17.084070 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:17.084037    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"]
Nov 03 08:03:17.124107 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:03:17.124073    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f1344d_5891_41db_9037_7e6f84a75530.slice/crio-46e2524e30105d0283d5276298958de5c18d10dd772809d9b6126f7bfcdacb6f WatchSource:0}: Error finding container 46e2524e30105d0283d5276298958de5c18d10dd772809d9b6126f7bfcdacb6f: Status 404 returned error can't find the container with id 46e2524e30105d0283d5276298958de5c18d10dd772809d9b6126f7bfcdacb6f
Nov 03 08:03:17.587123 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:17.587087    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s" event={"ID":"15f1344d-5891-41db-9037-7e6f84a75530","Type":"ContainerStarted","Data":"d3eacd4b79839b8b34a989b913c3f970efdeb17c605c39aac40de15bcafc317f"}
Nov 03 08:03:17.587561 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:17.587163    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s" event={"ID":"15f1344d-5891-41db-9037-7e6f84a75530","Type":"ContainerStarted","Data":"46e2524e30105d0283d5276298958de5c18d10dd772809d9b6126f7bfcdacb6f"}
Nov 03 08:03:17.707014 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:17.706905    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s" podStartSLOduration=1.706891686 podStartE2EDuration="1.706891686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:17.706214967 +0000 UTC m=+3850.038583962" watchObservedRunningTime="2026-11-03 08:03:17.706891686 +0000 UTC m=+3850.039260695"
Nov 03 08:03:18.593627 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:18.593589    4470 generic.go:356] "Generic (PLEG): container finished" podID="15f1344d-5891-41db-9037-7e6f84a75530" containerID="d3eacd4b79839b8b34a989b913c3f970efdeb17c605c39aac40de15bcafc317f" exitCode=0
Nov 03 08:03:18.594026 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:18.593674    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s" event={"ID":"15f1344d-5891-41db-9037-7e6f84a75530","Type":"ContainerDied","Data":"d3eacd4b79839b8b34a989b913c3f970efdeb17c605c39aac40de15bcafc317f"}
Nov 03 08:03:19.741811 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:19.741778    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:19.847148 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:19.847108    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f1344d-5891-41db-9037-7e6f84a75530-kube-api-access-c5bc9\" (UniqueName: \"kubernetes.io/projected/15f1344d-5891-41db-9037-7e6f84a75530-kube-api-access-c5bc9\") pod \"15f1344d-5891-41db-9037-7e6f84a75530\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") "
Nov 03 08:03:19.847318 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:19.847179    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-openshift-ca\") pod \"15f1344d-5891-41db-9037-7e6f84a75530\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") "
Nov 03 08:03:19.847318 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:19.847229    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-xks-ca\") pod \"15f1344d-5891-41db-9037-7e6f84a75530\" (UID: \"15f1344d-5891-41db-9037-7e6f84a75530\") "
Nov 03 08:03:19.847611 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:19.847576    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-openshift-ca" pod "15f1344d-5891-41db-9037-7e6f84a75530" (UID: "15f1344d-5891-41db-9037-7e6f84a75530"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:03:19.849546 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:19.849477    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-xks-ca" pod "15f1344d-5891-41db-9037-7e6f84a75530" (UID: "15f1344d-5891-41db-9037-7e6f84a75530"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:03:19.849652 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:19.849608    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f1344d-5891-41db-9037-7e6f84a75530-kube-api-access-c5bc9" pod "15f1344d-5891-41db-9037-7e6f84a75530" (UID: "15f1344d-5891-41db-9037-7e6f84a75530"). InnerVolumeSpecName "kube-api-access-c5bc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:03:19.948929 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:19.948886    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5bc9\" (UniqueName: \"kubernetes.io/projected/15f1344d-5891-41db-9037-7e6f84a75530-kube-api-access-c5bc9\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:03:19.948929 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:19.948922    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:03:19.948929 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:19.948933    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f1344d-5891-41db-9037-7e6f84a75530-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:03:20.605289 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:20.605257    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"
Nov 03 08:03:20.605289 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:20.605275    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s" event={"ID":"15f1344d-5891-41db-9037-7e6f84a75530","Type":"ContainerDied","Data":"46e2524e30105d0283d5276298958de5c18d10dd772809d9b6126f7bfcdacb6f"}
Nov 03 08:03:20.605578 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:20.605335    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46e2524e30105d0283d5276298958de5c18d10dd772809d9b6126f7bfcdacb6f"
Nov 03 08:03:27.910459 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:27.910415    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"]
Nov 03 08:03:28.021669 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:28.021636    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"]
Nov 03 08:03:28.072108 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:28.072067    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"]
Nov 03 08:03:28.074148 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:28.074113    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f8ee8b5-cb782"]
Nov 03 08:03:29.805266 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:29.805230    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff340ce-5865-4678-8439-aedb28246f6b" path="/var/lib/kubelet/pods/0ff340ce-5865-4678-8439-aedb28246f6b/volumes"
Nov 03 08:03:32.911504 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:32.911460    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm" podUID="f6462404-8ce7-4ce9-abab-5471fb3f0275" containerName="payload-processing" containerID="cri-o://8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3" gracePeriod=25
Nov 03 08:03:33.022014 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.021971    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw" podUID="3fb568ca-b9e1-4037-83af-998712ef7ca1" containerName="payload-pre-processing" containerID="cri-o://f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2" gracePeriod=25
Nov 03 08:03:33.168441 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.168382    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:03:33.282033 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.282010    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:03:33.293301 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.293269    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6462404-8ce7-4ce9-abab-5471fb3f0275-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6462404-8ce7-4ce9-abab-5471fb3f0275-plugins-config-volume\") pod \"f6462404-8ce7-4ce9-abab-5471fb3f0275\" (UID: \"f6462404-8ce7-4ce9-abab-5471fb3f0275\") "
Nov 03 08:03:33.293442 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.293358    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6462404-8ce7-4ce9-abab-5471fb3f0275-kube-api-access-m9zth\" (UniqueName: \"kubernetes.io/projected/f6462404-8ce7-4ce9-abab-5471fb3f0275-kube-api-access-m9zth\") pod \"f6462404-8ce7-4ce9-abab-5471fb3f0275\" (UID: \"f6462404-8ce7-4ce9-abab-5471fb3f0275\") "
Nov 03 08:03:33.293748 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.293719    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6462404-8ce7-4ce9-abab-5471fb3f0275-plugins-config-volume" pod "f6462404-8ce7-4ce9-abab-5471fb3f0275" (UID: "f6462404-8ce7-4ce9-abab-5471fb3f0275"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:03:33.293898 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.293829    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6462404-8ce7-4ce9-abab-5471fb3f0275-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:03:33.296185 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.296162    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6462404-8ce7-4ce9-abab-5471fb3f0275-kube-api-access-m9zth" pod "f6462404-8ce7-4ce9-abab-5471fb3f0275" (UID: "f6462404-8ce7-4ce9-abab-5471fb3f0275"). InnerVolumeSpecName "kube-api-access-m9zth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:03:33.394949 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.394917    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb568ca-b9e1-4037-83af-998712ef7ca1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb568ca-b9e1-4037-83af-998712ef7ca1-plugins-config-volume\") pod \"3fb568ca-b9e1-4037-83af-998712ef7ca1\" (UID: \"3fb568ca-b9e1-4037-83af-998712ef7ca1\") "
Nov 03 08:03:33.395121 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.395026    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb568ca-b9e1-4037-83af-998712ef7ca1-kube-api-access-zz429\" (UniqueName: \"kubernetes.io/projected/3fb568ca-b9e1-4037-83af-998712ef7ca1-kube-api-access-zz429\") pod \"3fb568ca-b9e1-4037-83af-998712ef7ca1\" (UID: \"3fb568ca-b9e1-4037-83af-998712ef7ca1\") "
Nov 03 08:03:33.395269 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.395256    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9zth\" (UniqueName: \"kubernetes.io/projected/f6462404-8ce7-4ce9-abab-5471fb3f0275-kube-api-access-m9zth\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:03:33.395329 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.395308    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb568ca-b9e1-4037-83af-998712ef7ca1-plugins-config-volume" pod "3fb568ca-b9e1-4037-83af-998712ef7ca1" (UID: "3fb568ca-b9e1-4037-83af-998712ef7ca1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:03:33.397195 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.397175    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb568ca-b9e1-4037-83af-998712ef7ca1-kube-api-access-zz429" pod "3fb568ca-b9e1-4037-83af-998712ef7ca1" (UID: "3fb568ca-b9e1-4037-83af-998712ef7ca1"). InnerVolumeSpecName "kube-api-access-zz429". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:03:33.496020 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.495937    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz429\" (UniqueName: \"kubernetes.io/projected/3fb568ca-b9e1-4037-83af-998712ef7ca1-kube-api-access-zz429\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:03:33.496020 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.495968    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb568ca-b9e1-4037-83af-998712ef7ca1-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:03:33.672849 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.672811    4470 generic.go:356] "Generic (PLEG): container finished" podID="f6462404-8ce7-4ce9-abab-5471fb3f0275" containerID="8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3" exitCode=0
Nov 03 08:03:33.672849 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.672854    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"
Nov 03 08:03:33.673120 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.672892    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm" event={"ID":"f6462404-8ce7-4ce9-abab-5471fb3f0275","Type":"ContainerDied","Data":"8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3"}
Nov 03 08:03:33.673120 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.672948    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm" event={"ID":"f6462404-8ce7-4ce9-abab-5471fb3f0275","Type":"ContainerDied","Data":"88c349eaaa9a84b17cee272f71d2fe7ae073694e55350f171b5ac5e54fdd81c7"}
Nov 03 08:03:33.673120 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.672972    4470 scope.go:122] "RemoveContainer" containerID="8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3"
Nov 03 08:03:33.674412 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.674393    4470 generic.go:356] "Generic (PLEG): container finished" podID="3fb568ca-b9e1-4037-83af-998712ef7ca1" containerID="f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2" exitCode=0
Nov 03 08:03:33.674706 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.674425    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw" event={"ID":"3fb568ca-b9e1-4037-83af-998712ef7ca1","Type":"ContainerDied","Data":"f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2"}
Nov 03 08:03:33.674706 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.674484    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw" event={"ID":"3fb568ca-b9e1-4037-83af-998712ef7ca1","Type":"ContainerDied","Data":"13310badbb8c4421dbde922e3a306c305924be707dba398fe065e57be3aedc8c"}
Nov 03 08:03:33.674706 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.674434    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"
Nov 03 08:03:33.682055 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.682038    4470 scope.go:122] "RemoveContainer" containerID="8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3"
Nov 03 08:03:33.682400 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:03:33.682374    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3\": container with ID starting with 8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3 not found: ID does not exist" containerID="8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3"
Nov 03 08:03:33.682522 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.682410    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3"} err="failed to get container status \"8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3\": rpc error: code = NotFound desc = could not find container \"8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3\": container with ID starting with 8e8250691e3c259f6d1f820919072a4d0df5acd193b6bb60743d2196d7caaaa3 not found: ID does not exist"
Nov 03 08:03:33.682522 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.682434    4470 scope.go:122] "RemoveContainer" containerID="f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2"
Nov 03 08:03:33.690863 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.690848    4470 scope.go:122] "RemoveContainer" containerID="f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2"
Nov 03 08:03:33.691161 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:03:33.691124    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2\": container with ID starting with f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2 not found: ID does not exist" containerID="f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2"
Nov 03 08:03:33.691220 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.691170    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2"} err="failed to get container status \"f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2\": rpc error: code = NotFound desc = could not find container \"f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2\": container with ID starting with f2473c21425b294de58c25079a341fab04cecd5c21d2b0887ed234748b5461a2 not found: ID does not exist"
Nov 03 08:03:33.846416 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.846385    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"]
Nov 03 08:03:33.897078 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.897049    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm"]
Nov 03 08:03:33.967029 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.966997    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"]
Nov 03 08:03:33.972727 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:33.972698    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw"]
Nov 03 08:03:34.759154 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:03:34.759091    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:03:35.805324 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:35.805290    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb568ca-b9e1-4037-83af-998712ef7ca1" path="/var/lib/kubelet/pods/3fb568ca-b9e1-4037-83af-998712ef7ca1/volumes"
Nov 03 08:03:35.805730 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:35.805677    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6462404-8ce7-4ce9-abab-5471fb3f0275" path="/var/lib/kubelet/pods/f6462404-8ce7-4ce9-abab-5471fb3f0275/volumes"
Nov 03 08:03:38.761918 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:38.761885    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"]
Nov 03 08:03:38.812522 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:38.812492    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b435d8-rxq7s"]
Nov 03 08:03:39.805324 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:39.805289    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f1344d-5891-41db-9037-7e6f84a75530" path="/var/lib/kubelet/pods/15f1344d-5891-41db-9037-7e6f84a75530/volumes"
Nov 03 08:03:53.417008 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.416972    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"]
Nov 03 08:03:53.417449 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.417401    4470 cpu_manager.go:414] "containerMap: removing container" podUID="f6462404-8ce7-4ce9-abab-5471fb3f0275" containerName="payload-processing"
Nov 03 08:03:53.417449 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.417412    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6462404-8ce7-4ce9-abab-5471fb3f0275" containerName="payload-processing"
Nov 03 08:03:53.417449 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.417424    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3fb568ca-b9e1-4037-83af-998712ef7ca1" containerName="payload-pre-processing"
Nov 03 08:03:53.417449 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.417430    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb568ca-b9e1-4037-83af-998712ef7ca1" containerName="payload-pre-processing"
Nov 03 08:03:53.417449 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.417440    4470 cpu_manager.go:414] "containerMap: removing container" podUID="15f1344d-5891-41db-9037-7e6f84a75530" containerName="revoke-keys"
Nov 03 08:03:53.417449 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.417446    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f1344d-5891-41db-9037-7e6f84a75530" containerName="revoke-keys"
Nov 03 08:03:53.417668 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.417514    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f1344d-5891-41db-9037-7e6f84a75530" containerName="revoke-keys"
Nov 03 08:03:53.417668 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.417529    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6462404-8ce7-4ce9-abab-5471fb3f0275" containerName="payload-processing"
Nov 03 08:03:53.417668 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.417536    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb568ca-b9e1-4037-83af-998712ef7ca1" containerName="payload-pre-processing"
Nov 03 08:03:53.420807 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.420789    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.480514 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.480481    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"]
Nov 03 08:03:53.494984 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.494951    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-home\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.495178 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.494990    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-dshm\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.495178 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.495022    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-tmp-dir\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.495276 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.495168    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrw7\" (UniqueName: \"kubernetes.io/projected/58454c33-23c8-4f80-9e9a-e125b05a7e14-kube-api-access-lcrw7\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.495341 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.495271    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58454c33-23c8-4f80-9e9a-e125b05a7e14-tls-certs\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.495341 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.495296    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-model-cache\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.528216 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.528177    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0730fa14-kserve-self-signed-certs\""
Nov 03 08:03:53.596679 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.596645    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-home\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.596679 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.596681    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-dshm\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.596971 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.596702    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-tmp-dir\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.596971 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.596721    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrw7\" (UniqueName: \"kubernetes.io/projected/58454c33-23c8-4f80-9e9a-e125b05a7e14-kube-api-access-lcrw7\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.596971 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.596779    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58454c33-23c8-4f80-9e9a-e125b05a7e14-tls-certs\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.596971 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.596795    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-model-cache\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.597240 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.597104    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-tmp-dir\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.597240 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.597173    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-home\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.597240 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.597221    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-model-cache\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.599242 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.599218    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-dshm\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.599510 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.599492    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58454c33-23c8-4f80-9e9a-e125b05a7e14-tls-certs\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.661638 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.661610    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrw7\" (UniqueName: \"kubernetes.io/projected/58454c33-23c8-4f80-9e9a-e125b05a7e14-kube-api-access-lcrw7\") pod \"e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") " pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:53.738108 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:53.738012    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:03:54.128550 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:54.128519    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"]
Nov 03 08:03:54.129845 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:03:54.129819    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58454c33_23c8_4f80_9e9a_e125b05a7e14.slice/crio-af1c0946e5f10a82b95ad1766d1583de9179367228e56b025d176807a0e8b8c2 WatchSource:0}: Error finding container af1c0946e5f10a82b95ad1766d1583de9179367228e56b025d176807a0e8b8c2: Status 404 returned error can't find the container with id af1c0946e5f10a82b95ad1766d1583de9179367228e56b025d176807a0e8b8c2
Nov 03 08:03:54.780529 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:54.780487    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88" event={"ID":"58454c33-23c8-4f80-9e9a-e125b05a7e14","Type":"ContainerStarted","Data":"2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0"}
Nov 03 08:03:54.780529 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:54.780536    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88" event={"ID":"58454c33-23c8-4f80-9e9a-e125b05a7e14","Type":"ContainerStarted","Data":"af1c0946e5f10a82b95ad1766d1583de9179367228e56b025d176807a0e8b8c2"}
Nov 03 08:03:54.902705 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:54.902646    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88" podStartSLOduration=1.902631506 podStartE2EDuration="1.902631506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:54.901331349 +0000 UTC m=+3887.233700345" watchObservedRunningTime="2026-11-03 08:03:54.902631506 +0000 UTC m=+3887.235000500"
Nov 03 08:03:55.738999 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:03:55.738964    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:04:03.739067 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:04:03.739021    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:04:03.751818 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:04:03.751786    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:04:03.873790 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:04:03.873760    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:04:04.758594 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:04:04.758560    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:04:05.055902 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:04:05.055822    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"]
Nov 03 08:04:05.106652 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:04:05.106617    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5hjk9"]
Nov 03 08:04:05.805064 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:04:05.805021    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f347c3-e6a6-4ffe-89d4-668f90243806" path="/var/lib/kubelet/pods/b0f347c3-e6a6-4ffe-89d4-668f90243806/volumes"
Nov 03 08:04:34.758539 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:04:34.758457    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:05:04.759761 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:05:04.759723    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:05:09.907834 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:09.907800    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"]
Nov 03 08:05:09.911467 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:09.911450    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:09.970517 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:09.970484    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"]
Nov 03 08:05:10.018713 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.018677    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:05:10.018713 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.018694    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:05:10.018713 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.018715    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:05:10.019017 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.018757    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 08:05:10.021216 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.021194    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qsz\" (UniqueName: \"kubernetes.io/projected/2fb59133-ab85-4c88-bc00-68bd0b487a50-kube-api-access-65qsz\") pod \"maas-api-revoke-sub-452b6d7b-bhqcz\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:10.021291 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.021234    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bhqcz\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:10.021412 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.021391    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bhqcz\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:10.122942 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.122906    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qsz\" (UniqueName: \"kubernetes.io/projected/2fb59133-ab85-4c88-bc00-68bd0b487a50-kube-api-access-65qsz\") pod \"maas-api-revoke-sub-452b6d7b-bhqcz\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:10.123170 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.122954    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bhqcz\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:10.123170 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.123043    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bhqcz\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:10.123779 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.123754    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bhqcz\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:10.125493 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.125474    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bhqcz\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:10.187863 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.187795    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qsz\" (UniqueName: \"kubernetes.io/projected/2fb59133-ab85-4c88-bc00-68bd0b487a50-kube-api-access-65qsz\") pod \"maas-api-revoke-sub-452b6d7b-bhqcz\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:10.223814 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.223785    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:10.602703 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:10.602679    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"]
Nov 03 08:05:10.604267 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:05:10.604238    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb59133_ab85_4c88_bc00_68bd0b487a50.slice/crio-8167c4d5e935251f0b29b9799e741ef9dbf6833749fc0edcccd6154f893206e6 WatchSource:0}: Error finding container 8167c4d5e935251f0b29b9799e741ef9dbf6833749fc0edcccd6154f893206e6: Status 404 returned error can't find the container with id 8167c4d5e935251f0b29b9799e741ef9dbf6833749fc0edcccd6154f893206e6
Nov 03 08:05:11.234532 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:11.234433    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz" event={"ID":"2fb59133-ab85-4c88-bc00-68bd0b487a50","Type":"ContainerStarted","Data":"8f4a5e43a50e9f7dd0aa9dc9af699ffca163816ef9ffedb1b7e3ba45f38c10e9"}
Nov 03 08:05:11.234532 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:11.234499    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz" event={"ID":"2fb59133-ab85-4c88-bc00-68bd0b487a50","Type":"ContainerStarted","Data":"8167c4d5e935251f0b29b9799e741ef9dbf6833749fc0edcccd6154f893206e6"}
Nov 03 08:05:11.354942 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:11.354895    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz" podStartSLOduration=2.354880611 podStartE2EDuration="2.354880611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:11.354695336 +0000 UTC m=+3963.687064330" watchObservedRunningTime="2026-11-03 08:05:11.354880611 +0000 UTC m=+3963.687249605"
Nov 03 08:05:12.244018 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:12.243978    4470 generic.go:356] "Generic (PLEG): container finished" podID="2fb59133-ab85-4c88-bc00-68bd0b487a50" containerID="8f4a5e43a50e9f7dd0aa9dc9af699ffca163816ef9ffedb1b7e3ba45f38c10e9" exitCode=0
Nov 03 08:05:12.244580 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:12.244025    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz" event={"ID":"2fb59133-ab85-4c88-bc00-68bd0b487a50","Type":"ContainerDied","Data":"8f4a5e43a50e9f7dd0aa9dc9af699ffca163816ef9ffedb1b7e3ba45f38c10e9"}
Nov 03 08:05:14.093394 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.093360    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:14.138160 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.136718    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb59133-ab85-4c88-bc00-68bd0b487a50-kube-api-access-65qsz\" (UniqueName: \"kubernetes.io/projected/2fb59133-ab85-4c88-bc00-68bd0b487a50-kube-api-access-65qsz\") pod \"2fb59133-ab85-4c88-bc00-68bd0b487a50\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") "
Nov 03 08:05:14.138160 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.136868    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-openshift-ca\") pod \"2fb59133-ab85-4c88-bc00-68bd0b487a50\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") "
Nov 03 08:05:14.138160 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.136933    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-xks-ca\") pod \"2fb59133-ab85-4c88-bc00-68bd0b487a50\" (UID: \"2fb59133-ab85-4c88-bc00-68bd0b487a50\") "
Nov 03 08:05:14.139912 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.138848    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-openshift-ca" pod "2fb59133-ab85-4c88-bc00-68bd0b487a50" (UID: "2fb59133-ab85-4c88-bc00-68bd0b487a50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:05:14.141787 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.141745    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-xks-ca" pod "2fb59133-ab85-4c88-bc00-68bd0b487a50" (UID: "2fb59133-ab85-4c88-bc00-68bd0b487a50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:05:14.144386 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.144349    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb59133-ab85-4c88-bc00-68bd0b487a50-kube-api-access-65qsz" pod "2fb59133-ab85-4c88-bc00-68bd0b487a50" (UID: "2fb59133-ab85-4c88-bc00-68bd0b487a50"). InnerVolumeSpecName "kube-api-access-65qsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:05:14.240159 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.238858    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qsz\" (UniqueName: \"kubernetes.io/projected/2fb59133-ab85-4c88-bc00-68bd0b487a50-kube-api-access-65qsz\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:14.240159 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.238901    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:14.240159 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.238926    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb59133-ab85-4c88-bc00-68bd0b487a50-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:14.395164 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.393534    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz" event={"ID":"2fb59133-ab85-4c88-bc00-68bd0b487a50","Type":"ContainerDied","Data":"8167c4d5e935251f0b29b9799e741ef9dbf6833749fc0edcccd6154f893206e6"}
Nov 03 08:05:14.395164 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.393601    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8167c4d5e935251f0b29b9799e741ef9dbf6833749fc0edcccd6154f893206e6"
Nov 03 08:05:14.395164 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.393671    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"
Nov 03 08:05:14.433720 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.433231    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:05:14.433720 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.433290    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw" podUID="cd812ea7-1e46-432d-acf9-ff1081a5863f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:14.562172 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.561250    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:05:14.562172 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.561335    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" podUID="f9b225a1-0e3c-48c9-9997-46b62edb5072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:14.919160 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.918250    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:05:14.919160 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:14.918337    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" podUID="63e33006-3dbb-4d05-a871-af997921a1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:15.486547 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:15.486510    4470 scope.go:122] "RemoveContainer" containerID="898664b5c9b2561ef2613ad8aec5011be28e894a1241fd68dd7ca0de61e5378c"
Nov 03 08:05:15.496893 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:15.496866    4470 scope.go:122] "RemoveContainer" containerID="c15f71355129d2db3da602b469c9f8a2a7fed1c72b1a1ca349768145fa64478c"
Nov 03 08:05:15.505917 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:15.505884    4470 scope.go:122] "RemoveContainer" containerID="b2d9aff1ec30ee6d5dba809318d56dff525094e5c1659687b2391c3e92afe506"
Nov 03 08:05:22.849287 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:22.849254    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"]
Nov 03 08:05:29.583159 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:29.582261    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:05:29.583159 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:29.582348    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb" podUID="f9b225a1-0e3c-48c9-9997-46b62edb5072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:29.893219 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:29.890043    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:05:29.893219 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:29.890173    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m" podUID="63e33006-3dbb-4d05-a871-af997921a1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:34.760119 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:05:34.760084    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:05:37.870057 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:37.870012    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88" podUID="58454c33-23c8-4f80-9e9a-e125b05a7e14" containerName="main" containerID="cri-o://2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0" gracePeriod=45
Nov 03 08:05:37.874361 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:37.874323    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88" podUID="58454c33-23c8-4f80-9e9a-e125b05a7e14" containerName="main" probeResult="failure" output="Get \"https://10.134.1.16:8000/ready\": read tcp 10.134.0.2:37598->10.134.1.16:8000: read: connection reset by peer"
Nov 03 08:05:38.117900 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.117871    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:05:38.255977 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.255897    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-home\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-home\") pod \"58454c33-23c8-4f80-9e9a-e125b05a7e14\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") "
Nov 03 08:05:38.255977 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.255936    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-tmp-dir\") pod \"58454c33-23c8-4f80-9e9a-e125b05a7e14\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") "
Nov 03 08:05:38.256229 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.256003    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-model-cache\") pod \"58454c33-23c8-4f80-9e9a-e125b05a7e14\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") "
Nov 03 08:05:38.256229 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.256049    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58454c33-23c8-4f80-9e9a-e125b05a7e14-tls-certs\" (UniqueName: \"kubernetes.io/secret/58454c33-23c8-4f80-9e9a-e125b05a7e14-tls-certs\") pod \"58454c33-23c8-4f80-9e9a-e125b05a7e14\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") "
Nov 03 08:05:38.256229 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.256080    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58454c33-23c8-4f80-9e9a-e125b05a7e14-kube-api-access-lcrw7\" (UniqueName: \"kubernetes.io/projected/58454c33-23c8-4f80-9e9a-e125b05a7e14-kube-api-access-lcrw7\") pod \"58454c33-23c8-4f80-9e9a-e125b05a7e14\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") "
Nov 03 08:05:38.256229 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.256115    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-dshm\") pod \"58454c33-23c8-4f80-9e9a-e125b05a7e14\" (UID: \"58454c33-23c8-4f80-9e9a-e125b05a7e14\") "
Nov 03 08:05:38.256229 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.256187    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-home" pod "58454c33-23c8-4f80-9e9a-e125b05a7e14" (UID: "58454c33-23c8-4f80-9e9a-e125b05a7e14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:05:38.256531 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.256270    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-model-cache" pod "58454c33-23c8-4f80-9e9a-e125b05a7e14" (UID: "58454c33-23c8-4f80-9e9a-e125b05a7e14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:05:38.256531 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.256285    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-tmp-dir" pod "58454c33-23c8-4f80-9e9a-e125b05a7e14" (UID: "58454c33-23c8-4f80-9e9a-e125b05a7e14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:05:38.256641 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.256548    4470 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-home\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:38.256641 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.256565    4470 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-tmp-dir\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:38.256641 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.256574    4470 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-model-cache\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:38.258569 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.258545    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58454c33-23c8-4f80-9e9a-e125b05a7e14-tls-certs" pod "58454c33-23c8-4f80-9e9a-e125b05a7e14" (UID: "58454c33-23c8-4f80-9e9a-e125b05a7e14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:05:38.258676 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.258621    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58454c33-23c8-4f80-9e9a-e125b05a7e14-kube-api-access-lcrw7" pod "58454c33-23c8-4f80-9e9a-e125b05a7e14" (UID: "58454c33-23c8-4f80-9e9a-e125b05a7e14"). InnerVolumeSpecName "kube-api-access-lcrw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:05:38.258676 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.258631    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-dshm" pod "58454c33-23c8-4f80-9e9a-e125b05a7e14" (UID: "58454c33-23c8-4f80-9e9a-e125b05a7e14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:05:38.357227 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.357195    4470 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58454c33-23c8-4f80-9e9a-e125b05a7e14-tls-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:38.357227 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.357227    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrw7\" (UniqueName: \"kubernetes.io/projected/58454c33-23c8-4f80-9e9a-e125b05a7e14-kube-api-access-lcrw7\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:38.357227 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.357237    4470 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58454c33-23c8-4f80-9e9a-e125b05a7e14-dshm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:38.550951 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.550865    4470 generic.go:356] "Generic (PLEG): container finished" podID="58454c33-23c8-4f80-9e9a-e125b05a7e14" containerID="2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0" exitCode=0
Nov 03 08:05:38.550951 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.550906    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"
Nov 03 08:05:38.551179 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.550957    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88" event={"ID":"58454c33-23c8-4f80-9e9a-e125b05a7e14","Type":"ContainerDied","Data":"2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0"}
Nov 03 08:05:38.551179 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.550997    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88" event={"ID":"58454c33-23c8-4f80-9e9a-e125b05a7e14","Type":"ContainerDied","Data":"af1c0946e5f10a82b95ad1766d1583de9179367228e56b025d176807a0e8b8c2"}
Nov 03 08:05:38.551179 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.551018    4470 scope.go:122] "RemoveContainer" containerID="2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0"
Nov 03 08:05:38.560833 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.560812    4470 scope.go:122] "RemoveContainer" containerID="2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0"
Nov 03 08:05:38.561116 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:05:38.561097    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0\": container with ID starting with 2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0 not found: ID does not exist" containerID="2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0"
Nov 03 08:05:38.561223 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.561127    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0"} err="failed to get container status \"2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0\": rpc error: code = NotFound desc = could not find container \"2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0\": container with ID starting with 2411b29db7cd545517df13d4ba68f0fbb4922c4012d50d6cd9fdf81ddabfaea0 not found: ID does not exist"
Nov 03 08:05:38.723616 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.723577    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"]
Nov 03 08:05:38.729275 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:38.729252    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0730fa14-kserve-7d8f4bffb9-26t88"]
Nov 03 08:05:39.806107 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:39.806074    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58454c33-23c8-4f80-9e9a-e125b05a7e14" path="/var/lib/kubelet/pods/58454c33-23c8-4f80-9e9a-e125b05a7e14/volumes"
Nov 03 08:05:45.835605 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.835560    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"]
Nov 03 08:05:45.836063 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.836047    4470 cpu_manager.go:414] "containerMap: removing container" podUID="58454c33-23c8-4f80-9e9a-e125b05a7e14" containerName="main"
Nov 03 08:05:45.836125 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.836066    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58454c33-23c8-4f80-9e9a-e125b05a7e14" containerName="main"
Nov 03 08:05:45.836125 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.836089    4470 cpu_manager.go:414] "containerMap: removing container" podUID="2fb59133-ab85-4c88-bc00-68bd0b487a50" containerName="revoke-keys"
Nov 03 08:05:45.836125 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.836098    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb59133-ab85-4c88-bc00-68bd0b487a50" containerName="revoke-keys"
Nov 03 08:05:45.836257 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.836243    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb59133-ab85-4c88-bc00-68bd0b487a50" containerName="revoke-keys"
Nov 03 08:05:45.836302 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.836261    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="58454c33-23c8-4f80-9e9a-e125b05a7e14" containerName="main"
Nov 03 08:05:45.839342 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.839321    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:45.932221 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.932183    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xtm5j\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:45.932221 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.932222    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gwg6\" (UniqueName: \"kubernetes.io/projected/ea763a73-892b-4833-b03e-6a433cd39931-kube-api-access-7gwg6\") pod \"maas-api-revoke-sub-3e31a506-xtm5j\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:45.932509 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.932349    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xtm5j\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:45.945965 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.945939    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:05:45.946485 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.946465    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 08:05:45.946601 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.946491    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:05:45.946601 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.946542    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:05:45.948115 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:45.948097    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"]
Nov 03 08:05:46.033405 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:46.033364    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xtm5j\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:46.033594 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:46.033490    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xtm5j\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:46.033594 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:46.033518    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gwg6\" (UniqueName: \"kubernetes.io/projected/ea763a73-892b-4833-b03e-6a433cd39931-kube-api-access-7gwg6\") pod \"maas-api-revoke-sub-3e31a506-xtm5j\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:46.034200 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:46.034119    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xtm5j\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:46.036157 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:46.036103    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xtm5j\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:46.098188 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:46.098096    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gwg6\" (UniqueName: \"kubernetes.io/projected/ea763a73-892b-4833-b03e-6a433cd39931-kube-api-access-7gwg6\") pod \"maas-api-revoke-sub-3e31a506-xtm5j\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:46.153200 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:46.153168    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:46.530264 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:46.530174    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"]
Nov 03 08:05:46.602735 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:46.602695    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j" event={"ID":"ea763a73-892b-4833-b03e-6a433cd39931","Type":"ContainerStarted","Data":"61d2b24ce365320455f1beb32eb0e35519147d6b1a71881f5125c08527078d6d"}
Nov 03 08:05:47.662158 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:47.661556    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j" event={"ID":"ea763a73-892b-4833-b03e-6a433cd39931","Type":"ContainerStarted","Data":"a7b6470fdf279e6ac27ba2426ac0ce65171b31fd2fb83f6c891e1e1697163e5e"}
Nov 03 08:05:48.710211 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:48.702679    4470 generic.go:356] "Generic (PLEG): container finished" podID="ea763a73-892b-4833-b03e-6a433cd39931" containerID="a7b6470fdf279e6ac27ba2426ac0ce65171b31fd2fb83f6c891e1e1697163e5e" exitCode=0
Nov 03 08:05:48.710211 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:48.702797    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j" event={"ID":"ea763a73-892b-4833-b03e-6a433cd39931","Type":"ContainerDied","Data":"a7b6470fdf279e6ac27ba2426ac0ce65171b31fd2fb83f6c891e1e1697163e5e"}
Nov 03 08:05:49.841253 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:49.841227    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:49.920765 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:49.920731    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-openshift-ca\") pod \"ea763a73-892b-4833-b03e-6a433cd39931\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") "
Nov 03 08:05:49.920935 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:49.920821    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea763a73-892b-4833-b03e-6a433cd39931-kube-api-access-7gwg6\" (UniqueName: \"kubernetes.io/projected/ea763a73-892b-4833-b03e-6a433cd39931-kube-api-access-7gwg6\") pod \"ea763a73-892b-4833-b03e-6a433cd39931\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") "
Nov 03 08:05:49.920935 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:49.920885    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-xks-ca\") pod \"ea763a73-892b-4833-b03e-6a433cd39931\" (UID: \"ea763a73-892b-4833-b03e-6a433cd39931\") "
Nov 03 08:05:49.921097 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:49.921074    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-openshift-ca" pod "ea763a73-892b-4833-b03e-6a433cd39931" (UID: "ea763a73-892b-4833-b03e-6a433cd39931"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:05:49.921322 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:49.921298    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:49.922847 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:49.922822    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-xks-ca" pod "ea763a73-892b-4833-b03e-6a433cd39931" (UID: "ea763a73-892b-4833-b03e-6a433cd39931"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:05:49.922938 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:49.922905    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea763a73-892b-4833-b03e-6a433cd39931-kube-api-access-7gwg6" pod "ea763a73-892b-4833-b03e-6a433cd39931" (UID: "ea763a73-892b-4833-b03e-6a433cd39931"). InnerVolumeSpecName "kube-api-access-7gwg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:05:50.022879 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:50.022803    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gwg6\" (UniqueName: \"kubernetes.io/projected/ea763a73-892b-4833-b03e-6a433cd39931-kube-api-access-7gwg6\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:50.022879 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:50.022831    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea763a73-892b-4833-b03e-6a433cd39931-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:05:50.713537 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:50.713510    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"
Nov 03 08:05:50.713739 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:50.713510    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j" event={"ID":"ea763a73-892b-4833-b03e-6a433cd39931","Type":"ContainerDied","Data":"61d2b24ce365320455f1beb32eb0e35519147d6b1a71881f5125c08527078d6d"}
Nov 03 08:05:50.713739 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:05:50.713632    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61d2b24ce365320455f1beb32eb0e35519147d6b1a71881f5125c08527078d6d"
Nov 03 08:06:04.839734 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:06:04.839684    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:06:09.056513 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:09.056477    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"]
Nov 03 08:06:09.107436 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:09.107405    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cfmhw"]
Nov 03 08:06:09.805770 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:09.805729    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea87ef0d-bc6b-4046-b16e-e5eb953be964" path="/var/lib/kubelet/pods/ea87ef0d-bc6b-4046-b16e-e5eb953be964/volumes"
Nov 03 08:06:18.588904 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.588873    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"]
Nov 03 08:06:18.589339 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.589302    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ea763a73-892b-4833-b03e-6a433cd39931" containerName="revoke-keys"
Nov 03 08:06:18.589339 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.589314    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea763a73-892b-4833-b03e-6a433cd39931" containerName="revoke-keys"
Nov 03 08:06:18.589429 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.589405    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea763a73-892b-4833-b03e-6a433cd39931" containerName="revoke-keys"
Nov 03 08:06:18.593576 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.593558    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:18.651574 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.651544    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"]
Nov 03 08:06:18.672712 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.672664    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxntw\" (UniqueName: \"kubernetes.io/projected/21fe5435-c199-4819-b31d-b38afe394613-kube-api-access-jxntw\") pod \"maas-api-revoke-sub-1b06fd92-jx9qr\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:18.672939 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.672906    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21fe5435-c199-4819-b31d-b38afe394613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b06fd92-jx9qr\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:18.673025 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.672958    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21fe5435-c199-4819-b31d-b38afe394613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b06fd92-jx9qr\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:18.699501 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.699463    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:06:18.699690 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.699543    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:06:18.699690 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.699617    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\""
Nov 03 08:06:18.699690 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.699656    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:06:18.773779 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.773735    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21fe5435-c199-4819-b31d-b38afe394613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b06fd92-jx9qr\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:18.773991 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.773788    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21fe5435-c199-4819-b31d-b38afe394613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b06fd92-jx9qr\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:18.773991 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.773839    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxntw\" (UniqueName: \"kubernetes.io/projected/21fe5435-c199-4819-b31d-b38afe394613-kube-api-access-jxntw\") pod \"maas-api-revoke-sub-1b06fd92-jx9qr\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:18.774578 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.774557    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21fe5435-c199-4819-b31d-b38afe394613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b06fd92-jx9qr\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:18.776382 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.776358    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21fe5435-c199-4819-b31d-b38afe394613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b06fd92-jx9qr\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:18.838218 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.838184    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxntw\" (UniqueName: \"kubernetes.io/projected/21fe5435-c199-4819-b31d-b38afe394613-kube-api-access-jxntw\") pod \"maas-api-revoke-sub-1b06fd92-jx9qr\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:18.906022 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:18.905938    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:19.282101 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:19.282077    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"]
Nov 03 08:06:19.283698 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:06:19.283672    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21fe5435_c199_4819_b31d_b38afe394613.slice/crio-9f48f11bf13409efddc17a83a39ce112d91af172bcce8b53dff800b88a009eca WatchSource:0}: Error finding container 9f48f11bf13409efddc17a83a39ce112d91af172bcce8b53dff800b88a009eca: Status 404 returned error can't find the container with id 9f48f11bf13409efddc17a83a39ce112d91af172bcce8b53dff800b88a009eca
Nov 03 08:06:19.902827 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:19.902789    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr" event={"ID":"21fe5435-c199-4819-b31d-b38afe394613","Type":"ContainerStarted","Data":"0c3720d1362f153697a3ac45269b1c8c66e235b65aa3cd62bc982bd0ff9a2a61"}
Nov 03 08:06:19.903274 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:19.902854    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr" event={"ID":"21fe5435-c199-4819-b31d-b38afe394613","Type":"ContainerStarted","Data":"9f48f11bf13409efddc17a83a39ce112d91af172bcce8b53dff800b88a009eca"}
Nov 03 08:06:20.022704 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:20.022642    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr" podStartSLOduration=2.022625231 podStartE2EDuration="2.022625231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:20.021513623 +0000 UTC m=+4032.353882616" watchObservedRunningTime="2026-11-03 08:06:20.022625231 +0000 UTC m=+4032.354994226"
Nov 03 08:06:20.908483 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:20.908452    4470 generic.go:356] "Generic (PLEG): container finished" podID="21fe5435-c199-4819-b31d-b38afe394613" containerID="0c3720d1362f153697a3ac45269b1c8c66e235b65aa3cd62bc982bd0ff9a2a61" exitCode=0
Nov 03 08:06:20.908899 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:20.908532    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr" event={"ID":"21fe5435-c199-4819-b31d-b38afe394613","Type":"ContainerDied","Data":"0c3720d1362f153697a3ac45269b1c8c66e235b65aa3cd62bc982bd0ff9a2a61"}
Nov 03 08:06:22.055746 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.055718    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:22.107493 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.107461    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21fe5435-c199-4819-b31d-b38afe394613-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21fe5435-c199-4819-b31d-b38afe394613-maas-api-openshift-ca\") pod \"21fe5435-c199-4819-b31d-b38afe394613\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") "
Nov 03 08:06:22.107658 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.107642    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21fe5435-c199-4819-b31d-b38afe394613-kube-api-access-jxntw\" (UniqueName: \"kubernetes.io/projected/21fe5435-c199-4819-b31d-b38afe394613-kube-api-access-jxntw\") pod \"21fe5435-c199-4819-b31d-b38afe394613\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") "
Nov 03 08:06:22.107713 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.107700    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21fe5435-c199-4819-b31d-b38afe394613-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21fe5435-c199-4819-b31d-b38afe394613-maas-api-xks-ca\") pod \"21fe5435-c199-4819-b31d-b38afe394613\" (UID: \"21fe5435-c199-4819-b31d-b38afe394613\") "
Nov 03 08:06:22.107874 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.107853    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21fe5435-c199-4819-b31d-b38afe394613-maas-api-openshift-ca" pod "21fe5435-c199-4819-b31d-b38afe394613" (UID: "21fe5435-c199-4819-b31d-b38afe394613"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:06:22.108002 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.107981    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21fe5435-c199-4819-b31d-b38afe394613-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:06:22.109872 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.109849    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21fe5435-c199-4819-b31d-b38afe394613-maas-api-xks-ca" pod "21fe5435-c199-4819-b31d-b38afe394613" (UID: "21fe5435-c199-4819-b31d-b38afe394613"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:06:22.109981 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.109960    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21fe5435-c199-4819-b31d-b38afe394613-kube-api-access-jxntw" pod "21fe5435-c199-4819-b31d-b38afe394613" (UID: "21fe5435-c199-4819-b31d-b38afe394613"). InnerVolumeSpecName "kube-api-access-jxntw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:06:22.209318 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.209238    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxntw\" (UniqueName: \"kubernetes.io/projected/21fe5435-c199-4819-b31d-b38afe394613-kube-api-access-jxntw\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:06:22.209318 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.209268    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21fe5435-c199-4819-b31d-b38afe394613-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:06:22.919559 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.919524    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr" event={"ID":"21fe5435-c199-4819-b31d-b38afe394613","Type":"ContainerDied","Data":"9f48f11bf13409efddc17a83a39ce112d91af172bcce8b53dff800b88a009eca"}
Nov 03 08:06:22.919776 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.919566    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"
Nov 03 08:06:22.919776 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:22.919572    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f48f11bf13409efddc17a83a39ce112d91af172bcce8b53dff800b88a009eca"
Nov 03 08:06:34.758698 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:06:34.758659    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:06:38.053985 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.053949    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"]
Nov 03 08:06:38.111691 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.105047    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ngssg"]
Nov 03 08:06:38.829645 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.829609    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"]
Nov 03 08:06:38.830105 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.830089    4470 cpu_manager.go:414] "containerMap: removing container" podUID="21fe5435-c199-4819-b31d-b38afe394613" containerName="revoke-keys"
Nov 03 08:06:38.830197 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.830108    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21fe5435-c199-4819-b31d-b38afe394613" containerName="revoke-keys"
Nov 03 08:06:38.830240 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.830207    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="21fe5435-c199-4819-b31d-b38afe394613" containerName="revoke-keys"
Nov 03 08:06:38.833357 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.833343    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.894188 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.894119    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-podinfo\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.894188 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.894184    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d80b4d7-8674-481b-9e76-d034417a64c9-istiod-ca-cert\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.894441 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.894292    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-token\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.894441 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.894339    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46fgl\" (UniqueName: \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-kube-api-access-46fgl\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.894441 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.894408    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-socket\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.894441 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.894439    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-envoy\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.894610 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.894541    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-data\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.894610 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.894582    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-credential-socket\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.894685 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.894668    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-certs\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.903052 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.903034    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-16cc64e2-openshift-default-dockercfg-qtrbw\""
Nov 03 08:06:38.948167 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.948108    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"]
Nov 03 08:06:38.995598 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.995567    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-token\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.995803 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.995617    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46fgl\" (UniqueName: \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-kube-api-access-46fgl\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.995803 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.995731    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-socket\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.995803 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.995770    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-envoy\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.995985 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.995807    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-data\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.995985 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.995832    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-credential-socket\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.995985 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.995889    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-certs\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.996176 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.995990    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-podinfo\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.996176 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.996023    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d80b4d7-8674-481b-9e76-d034417a64c9-istiod-ca-cert\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.996289 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.996173    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-socket\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.996289 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.996273    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-data\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.996407 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.996327    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-certs\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.996635 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.996611    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-credential-socket\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.996818 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.996786    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d80b4d7-8674-481b-9e76-d034417a64c9-istiod-ca-cert\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.998365 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.998344    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-envoy\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:38.999206 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:38.999189    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-podinfo\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:39.061951 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:39.061917    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-token\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:39.062437 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:39.062082    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46fgl\" (UniqueName: \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-kube-api-access-46fgl\") pod \"e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") " pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:39.146246 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:39.146171    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:39.575913 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:39.575889    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"]
Nov 03 08:06:39.577901 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:06:39.577868    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d80b4d7_8674_481b_9e76_d034417a64c9.slice/crio-a18fa45c2a1b7030ae01a86f4a71d951e0dd713fb5e10713a4b01cad8a2c55e1 WatchSource:0}: Error finding container a18fa45c2a1b7030ae01a86f4a71d951e0dd713fb5e10713a4b01cad8a2c55e1: Status 404 returned error can't find the container with id a18fa45c2a1b7030ae01a86f4a71d951e0dd713fb5e10713a4b01cad8a2c55e1
Nov 03 08:06:39.580430 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:39.580398    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 08:06:39.580507 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:39.580463    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 08:06:39.580507 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:39.580491    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 08:06:39.806067 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:39.806030    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c" path="/var/lib/kubelet/pods/fb7b6cda-cf4e-4cbb-90a3-cfd6c92b204c/volumes"
Nov 03 08:06:40.004300 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:40.004219    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" event={"ID":"9d80b4d7-8674-481b-9e76-d034417a64c9","Type":"ContainerStarted","Data":"f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2"}
Nov 03 08:06:40.004300 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:40.004270    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" event={"ID":"9d80b4d7-8674-481b-9e76-d034417a64c9","Type":"ContainerStarted","Data":"a18fa45c2a1b7030ae01a86f4a71d951e0dd713fb5e10713a4b01cad8a2c55e1"}
Nov 03 08:06:40.130598 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:40.130552    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" podStartSLOduration=2.130539748 podStartE2EDuration="2.130539748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:40.129681172 +0000 UTC m=+4052.462050168" watchObservedRunningTime="2026-11-03 08:06:40.130539748 +0000 UTC m=+4052.462908742"
Nov 03 08:06:40.147157 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:40.147110    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:40.152385 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:40.152360    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:41.009416 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:41.009383    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:06:41.010391 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:06:41.010366    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:07:04.759739 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:07:04.759698    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:07:06.155013 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:06.154981    4470 generic.go:356] "Generic (PLEG): container finished" podID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerID="f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2" exitCode=0
Nov 03 08:07:06.155445 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:06.155059    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" event={"ID":"9d80b4d7-8674-481b-9e76-d034417a64c9","Type":"ContainerDied","Data":"f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2"}
Nov 03 08:07:06.155495 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:06.155461    4470 scope.go:122] "RemoveContainer" containerID="f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2"
Nov 03 08:07:06.156730 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:06.156702    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 08:07:06.156808 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:06.156772    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 08:07:06.156808 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:06.156802    4470 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 08:07:07.175356 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:07.175314    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" event={"ID":"9d80b4d7-8674-481b-9e76-d034417a64c9","Type":"ContainerStarted","Data":"428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67"}
Nov 03 08:07:08.171163 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:08.168889    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:07:08.171434 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:08.171403    4470 patch_prober.go:29] interesting pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 08:07:08.171502 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:08.171464    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 08:07:09.157160 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:09.149511    4470 patch_prober.go:29] interesting pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 08:07:09.157160 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:09.149571    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 08:07:10.161159 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:10.155482    4470 patch_prober.go:29] interesting pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 08:07:10.161159 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:10.155540    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 08:07:11.009461 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:11.009426    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:07:11.149496 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:11.149459    4470 patch_prober.go:29] interesting pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 08:07:11.149780 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:11.149751    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 08:07:12.147084 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:12.147049    4470 patch_prober.go:29] interesting pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 08:07:12.147638 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:12.147104    4470 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 08:07:13.150646 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:13.150607    4470 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:07:13.151747 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:13.151727    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:07:15.544076 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:15.544045    4470 scope.go:122] "RemoveContainer" containerID="2a60562cda192dfa388516438106f65aca0f4bcd7c3d3edde2c43511bfb26c35"
Nov 03 08:07:18.696779 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.696694    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"]
Nov 03 08:07:18.701705 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.701686    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:18.759120 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.759089    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"]
Nov 03 08:07:18.805971 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.805934    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:18.806186 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.806001    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:18.806186 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.806085    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz2nx\" (UniqueName: \"kubernetes.io/projected/a4fda601-efa7-464d-ad96-3f997a17fe56-kube-api-access-rz2nx\") pod \"maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:18.807237 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.807216    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:07:18.807404 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.807384    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55sgn\""
Nov 03 08:07:18.807500 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.807411    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:07:18.807500 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.807422    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:07:18.907551 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.907517    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:18.907735 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.907568    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:18.907735 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.907619    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz2nx\" (UniqueName: \"kubernetes.io/projected/a4fda601-efa7-464d-ad96-3f997a17fe56-kube-api-access-rz2nx\") pod \"maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:18.908210 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.908192    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:18.910068 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.910045    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:18.972070 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:18.972008    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz2nx\" (UniqueName: \"kubernetes.io/projected/a4fda601-efa7-464d-ad96-3f997a17fe56-kube-api-access-rz2nx\") pod \"maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:19.014878 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:19.014852    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:19.391209 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:19.391181    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"]
Nov 03 08:07:19.392943 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:07:19.392907    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4fda601_efa7_464d_ad96_3f997a17fe56.slice/crio-9e46f2f695083f896712cfb474cb1e5fa28d83b9beb0674c1f25b30b3a67a017 WatchSource:0}: Error finding container 9e46f2f695083f896712cfb474cb1e5fa28d83b9beb0674c1f25b30b3a67a017: Status 404 returned error can't find the container with id 9e46f2f695083f896712cfb474cb1e5fa28d83b9beb0674c1f25b30b3a67a017
Nov 03 08:07:20.244111 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:20.244080    4470 generic.go:356] "Generic (PLEG): container finished" podID="a4fda601-efa7-464d-ad96-3f997a17fe56" containerID="7d723a3712dd7af0cdb7ca8cdc16271ad7c874f537269671f5d9986699a389df" exitCode=0
Nov 03 08:07:20.244519 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:20.244167    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp" event={"ID":"a4fda601-efa7-464d-ad96-3f997a17fe56","Type":"ContainerDied","Data":"7d723a3712dd7af0cdb7ca8cdc16271ad7c874f537269671f5d9986699a389df"}
Nov 03 08:07:20.244519 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:20.244217    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp" event={"ID":"a4fda601-efa7-464d-ad96-3f997a17fe56","Type":"ContainerStarted","Data":"9e46f2f695083f896712cfb474cb1e5fa28d83b9beb0674c1f25b30b3a67a017"}
Nov 03 08:07:21.383933 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:21.383912    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:21.535651 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:21.535563    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4fda601-efa7-464d-ad96-3f997a17fe56-kube-api-access-rz2nx\" (UniqueName: \"kubernetes.io/projected/a4fda601-efa7-464d-ad96-3f997a17fe56-kube-api-access-rz2nx\") pod \"a4fda601-efa7-464d-ad96-3f997a17fe56\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") "
Nov 03 08:07:21.535651 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:21.535631    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-openshift-ca\") pod \"a4fda601-efa7-464d-ad96-3f997a17fe56\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") "
Nov 03 08:07:21.535890 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:21.535748    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-xks-ca\") pod \"a4fda601-efa7-464d-ad96-3f997a17fe56\" (UID: \"a4fda601-efa7-464d-ad96-3f997a17fe56\") "
Nov 03 08:07:21.535988 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:21.535965    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-openshift-ca" pod "a4fda601-efa7-464d-ad96-3f997a17fe56" (UID: "a4fda601-efa7-464d-ad96-3f997a17fe56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:21.536095 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:21.536075    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:21.537975 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:21.537949    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fda601-efa7-464d-ad96-3f997a17fe56-kube-api-access-rz2nx" pod "a4fda601-efa7-464d-ad96-3f997a17fe56" (UID: "a4fda601-efa7-464d-ad96-3f997a17fe56"). InnerVolumeSpecName "kube-api-access-rz2nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:21.538082 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:21.538011    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-xks-ca" pod "a4fda601-efa7-464d-ad96-3f997a17fe56" (UID: "a4fda601-efa7-464d-ad96-3f997a17fe56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:21.637450 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:21.637416    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fda601-efa7-464d-ad96-3f997a17fe56-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:21.637450 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:21.637446    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz2nx\" (UniqueName: \"kubernetes.io/projected/a4fda601-efa7-464d-ad96-3f997a17fe56-kube-api-access-rz2nx\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:22.255412 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:22.255326    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"
Nov 03 08:07:22.255412 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:22.255350    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp" event={"ID":"a4fda601-efa7-464d-ad96-3f997a17fe56","Type":"ContainerDied","Data":"9e46f2f695083f896712cfb474cb1e5fa28d83b9beb0674c1f25b30b3a67a017"}
Nov 03 08:07:22.255412 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:22.255403    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e46f2f695083f896712cfb474cb1e5fa28d83b9beb0674c1f25b30b3a67a017"
Nov 03 08:07:34.758634 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:07:34.758598    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:07:34.923267 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:34.923232    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"]
Nov 03 08:07:34.979074 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:34.974532    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cc64e2-7qsgp"]
Nov 03 08:07:35.805522 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:35.805489    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4fda601-efa7-464d-ad96-3f997a17fe56" path="/var/lib/kubelet/pods/a4fda601-efa7-464d-ad96-3f997a17fe56/volumes"
Nov 03 08:07:39.789697 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:39.789669    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"]
Nov 03 08:07:39.790152 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:39.789870    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy" containerID="cri-o://428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67" gracePeriod=30
Nov 03 08:07:43.153221 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:43.153188    4470 patch_prober.go:29] interesting pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:07:43.153625 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:43.153250    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:07:45.047618 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.047587    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:07:45.074808 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.074773    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-certs\") pod \"9d80b4d7-8674-481b-9e76-d034417a64c9\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") "
Nov 03 08:07:45.074989 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.074818    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-data\") pod \"9d80b4d7-8674-481b-9e76-d034417a64c9\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") "
Nov 03 08:07:45.074989 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.074871    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-token\" (UniqueName: \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-token\") pod \"9d80b4d7-8674-481b-9e76-d034417a64c9\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") "
Nov 03 08:07:45.074989 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.074903    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-envoy\") pod \"9d80b4d7-8674-481b-9e76-d034417a64c9\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") "
Nov 03 08:07:45.074989 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.074947    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d80b4d7-8674-481b-9e76-d034417a64c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d80b4d7-8674-481b-9e76-d034417a64c9-istiod-ca-cert\") pod \"9d80b4d7-8674-481b-9e76-d034417a64c9\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") "
Nov 03 08:07:45.075268 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075029    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-socket\") pod \"9d80b4d7-8674-481b-9e76-d034417a64c9\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") "
Nov 03 08:07:45.075268 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075070    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-credential-socket\") pod \"9d80b4d7-8674-481b-9e76-d034417a64c9\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") "
Nov 03 08:07:45.075268 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075101    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-podinfo\") pod \"9d80b4d7-8674-481b-9e76-d034417a64c9\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") "
Nov 03 08:07:45.075268 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075086    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-certs" pod "9d80b4d7-8674-481b-9e76-d034417a64c9" (UID: "9d80b4d7-8674-481b-9e76-d034417a64c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:07:45.075268 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075188    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-data" pod "9d80b4d7-8674-481b-9e76-d034417a64c9" (UID: "9d80b4d7-8674-481b-9e76-d034417a64c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:07:45.075268 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075200    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-kube-api-access-46fgl\" (UniqueName: \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-kube-api-access-46fgl\") pod \"9d80b4d7-8674-481b-9e76-d034417a64c9\" (UID: \"9d80b4d7-8674-481b-9e76-d034417a64c9\") "
Nov 03 08:07:45.075578 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075413    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-socket" pod "9d80b4d7-8674-481b-9e76-d034417a64c9" (UID: "9d80b4d7-8674-481b-9e76-d034417a64c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:07:45.075578 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075423    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-credential-socket" pod "9d80b4d7-8674-481b-9e76-d034417a64c9" (UID: "9d80b4d7-8674-481b-9e76-d034417a64c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:07:45.075578 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075533    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d80b4d7-8674-481b-9e76-d034417a64c9-istiod-ca-cert" pod "9d80b4d7-8674-481b-9e76-d034417a64c9" (UID: "9d80b4d7-8674-481b-9e76-d034417a64c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:45.075933 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075597    4470 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-certs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:45.075933 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075621    4470 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-data\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:45.075933 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075637    4470 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-workload-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:45.075933 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.075646    4470 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-credential-socket\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:45.077381 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.077354    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-envoy" pod "9d80b4d7-8674-481b-9e76-d034417a64c9" (UID: "9d80b4d7-8674-481b-9e76-d034417a64c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:07:45.077493 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.077400    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-podinfo" pod "9d80b4d7-8674-481b-9e76-d034417a64c9" (UID: "9d80b4d7-8674-481b-9e76-d034417a64c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:07:45.077493 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.077458    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-token" pod "9d80b4d7-8674-481b-9e76-d034417a64c9" (UID: "9d80b4d7-8674-481b-9e76-d034417a64c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:45.077598 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.077497    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-kube-api-access-46fgl" pod "9d80b4d7-8674-481b-9e76-d034417a64c9" (UID: "9d80b4d7-8674-481b-9e76-d034417a64c9"). InnerVolumeSpecName "kube-api-access-46fgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:45.176908 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.176875    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46fgl\" (UniqueName: \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-kube-api-access-46fgl\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:45.176908 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.176903    4470 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-token\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:45.176908 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.176916    4470 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-envoy\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:45.177202 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.176925    4470 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d80b4d7-8674-481b-9e76-d034417a64c9-istiod-ca-cert\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:45.177202 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.176934    4470 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d80b4d7-8674-481b-9e76-d034417a64c9-istio-podinfo\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:07:45.371955 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.371857    4470 generic.go:356] "Generic (PLEG): container finished" podID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerID="428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67" exitCode=0
Nov 03 08:07:45.371955 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.371905    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"
Nov 03 08:07:45.371955 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.371943    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" event={"ID":"9d80b4d7-8674-481b-9e76-d034417a64c9","Type":"ContainerDied","Data":"428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67"}
Nov 03 08:07:45.372252 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.371999    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q" event={"ID":"9d80b4d7-8674-481b-9e76-d034417a64c9","Type":"ContainerDied","Data":"a18fa45c2a1b7030ae01a86f4a71d951e0dd713fb5e10713a4b01cad8a2c55e1"}
Nov 03 08:07:45.372252 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.372021    4470 scope.go:122] "RemoveContainer" containerID="428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67"
Nov 03 08:07:45.382157 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.382118    4470 scope.go:122] "RemoveContainer" containerID="f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2"
Nov 03 08:07:45.391009 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.390982    4470 scope.go:122] "RemoveContainer" containerID="428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67"
Nov 03 08:07:45.391334 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:07:45.391314    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67\": container with ID starting with 428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67 not found: ID does not exist" containerID="428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67"
Nov 03 08:07:45.391388 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.391347    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67"} err="failed to get container status \"428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67\": rpc error: code = NotFound desc = could not find container \"428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67\": container with ID starting with 428ddceaad5435845f0c1a67a71673d52a2967bad39037590b1b98c70235ed67 not found: ID does not exist"
Nov 03 08:07:45.391388 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.391365    4470 scope.go:122] "RemoveContainer" containerID="f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2"
Nov 03 08:07:45.391620 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:07:45.391596    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2\": container with ID starting with f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2 not found: ID does not exist" containerID="f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2"
Nov 03 08:07:45.391660 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.391628    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2"} err="failed to get container status \"f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2\": rpc error: code = NotFound desc = could not find container \"f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2\": container with ID starting with f465f91be77a16bd2243cfd02aa681bf984ea929f07e41776fee7d60916e74e2 not found: ID does not exist"
Nov 03 08:07:45.542797 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.542760    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"]
Nov 03 08:07:45.548994 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.548970    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q"]
Nov 03 08:07:45.805350 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:45.805275    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" path="/var/lib/kubelet/pods/9d80b4d7-8674-481b-9e76-d034417a64c9/volumes"
Nov 03 08:07:49.370018 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.369981    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"]
Nov 03 08:07:49.370524 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.370411    4470 cpu_manager.go:414] "containerMap: removing container" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy"
Nov 03 08:07:49.370524 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.370425    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy"
Nov 03 08:07:49.370524 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.370458    4470 cpu_manager.go:414] "containerMap: removing container" podUID="a4fda601-efa7-464d-ad96-3f997a17fe56" containerName="revoke-keys"
Nov 03 08:07:49.370524 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.370463    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4fda601-efa7-464d-ad96-3f997a17fe56" containerName="revoke-keys"
Nov 03 08:07:49.370524 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.370524    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy"
Nov 03 08:07:49.370698 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.370536    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy"
Nov 03 08:07:49.370698 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.370544    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4fda601-efa7-464d-ad96-3f997a17fe56" containerName="revoke-keys"
Nov 03 08:07:49.374857 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.374840    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:07:49.417304 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.417272    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwxm\" (UniqueName: \"kubernetes.io/projected/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-kube-api-access-2fwxm\") pod \"payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5\" (UID: \"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:07:49.417528 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.417382    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5\" (UID: \"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:07:49.435072 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.435045    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"]
Nov 03 08:07:49.484788 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.484765    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-3ef065\""
Nov 03 08:07:49.512725 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.512695    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"]
Nov 03 08:07:49.513398 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.513372    4470 cpu_manager.go:414] "containerMap: removing container" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy"
Nov 03 08:07:49.513398 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.513397    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d80b4d7-8674-481b-9e76-d034417a64c9" containerName="istio-proxy"
Nov 03 08:07:49.517014 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.516805    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:07:49.518394 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.518370    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5\" (UID: \"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:07:49.518561 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.518519    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwxm\" (UniqueName: \"kubernetes.io/projected/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-kube-api-access-2fwxm\") pod \"payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5\" (UID: \"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:07:49.519339 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.519313    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5\" (UID: \"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:07:49.574228 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.574197    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-3ef065-dockercfg-9p4gs\""
Nov 03 08:07:49.574465 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.574442    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"]
Nov 03 08:07:49.582494 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.582462    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwxm\" (UniqueName: \"kubernetes.io/projected/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-kube-api-access-2fwxm\") pod \"payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5\" (UID: \"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:07:49.619234 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.619205    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65\" (UID: \"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:07:49.619386 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.619350    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54sg9\" (UniqueName: \"kubernetes.io/projected/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-kube-api-access-54sg9\") pod \"payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65\" (UID: \"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:07:49.687540 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.687467    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:07:49.720165 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.720119    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54sg9\" (UniqueName: \"kubernetes.io/projected/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-kube-api-access-54sg9\") pod \"payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65\" (UID: \"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:07:49.720300 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.720217    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65\" (UID: \"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:07:49.721104 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.721079    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65\" (UID: \"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:07:49.785504 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.785461    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54sg9\" (UniqueName: \"kubernetes.io/projected/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-kube-api-access-54sg9\") pod \"payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65\" (UID: \"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:07:49.832863 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:49.832831    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:07:50.060056 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:50.060033    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"]
Nov 03 08:07:50.061581 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:07:50.061555    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb910a0eb_dca1_4a2f_9e38_4d31fcf5a322.slice/crio-77d498b05fa8087ec2c1c6ef84cafeb73f995dcf2b0dde24b4b22be718fd10dc WatchSource:0}: Error finding container 77d498b05fa8087ec2c1c6ef84cafeb73f995dcf2b0dde24b4b22be718fd10dc: Status 404 returned error can't find the container with id 77d498b05fa8087ec2c1c6ef84cafeb73f995dcf2b0dde24b4b22be718fd10dc
Nov 03 08:07:50.209448 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:50.209421    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"]
Nov 03 08:07:50.210918 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:07:50.210889    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed4cdac_43d6_42a6_9cf7_99f9c2d8a011.slice/crio-af473a8ffc723cef8df94ca91011db5dd427ba4bd31d8153be5863643efd3226 WatchSource:0}: Error finding container af473a8ffc723cef8df94ca91011db5dd427ba4bd31d8153be5863643efd3226: Status 404 returned error can't find the container with id af473a8ffc723cef8df94ca91011db5dd427ba4bd31d8153be5863643efd3226
Nov 03 08:07:50.398190 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:50.398121    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65" event={"ID":"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011","Type":"ContainerStarted","Data":"40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68"}
Nov 03 08:07:50.398669 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:50.398209    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:07:50.398669 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:50.398232    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65" event={"ID":"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011","Type":"ContainerStarted","Data":"af473a8ffc723cef8df94ca91011db5dd427ba4bd31d8153be5863643efd3226"}
Nov 03 08:07:50.399788 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:50.399765    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5" event={"ID":"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322","Type":"ContainerStarted","Data":"134815f27889b00f380f39be37c99b37ac58e4b7605e82b28ee89d8a04851627"}
Nov 03 08:07:50.399906 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:50.399810    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:07:50.399906 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:50.399825    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5" event={"ID":"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322","Type":"ContainerStarted","Data":"77d498b05fa8087ec2c1c6ef84cafeb73f995dcf2b0dde24b4b22be718fd10dc"}
Nov 03 08:07:50.519376 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:50.519323    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65" podStartSLOduration=1.519308613 podStartE2EDuration="1.519308613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:50.518286536 +0000 UTC m=+4122.850655531" watchObservedRunningTime="2026-11-03 08:07:50.519308613 +0000 UTC m=+4122.851677606"
Nov 03 08:07:50.639842 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:07:50.639780    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5" podStartSLOduration=1.639759663 podStartE2EDuration="1.639759663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:50.639389525 +0000 UTC m=+4122.971758521" watchObservedRunningTime="2026-11-03 08:07:50.639759663 +0000 UTC m=+4122.972128658"
Nov 03 08:08:01.408359 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:08:01.408331    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:08:01.408744 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:08:01.408396    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:08:04.759169 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:08:04.759118    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:08:08.067261 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:08:08.067218    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"]
Nov 03 08:08:08.117890 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:08:08.117853    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48198fc4-hlzcj"]
Nov 03 08:08:09.804969 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:08:09.804935    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c931178-6eac-46b6-9c3e-308fe2e37e24" path="/var/lib/kubelet/pods/5c931178-6eac-46b6-9c3e-308fe2e37e24/volumes"
Nov 03 08:08:15.564909 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:08:15.564871    4470 scope.go:122] "RemoveContainer" containerID="76ccd1745ebdc624452c25327f07f1b70fe4297943b0ddd40ac829d3baa8ba33"
Nov 03 08:08:34.759682 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:08:34.759643    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:09:04.759244 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:09:04.759212    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:09:15.592175 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:15.592126    4470 scope.go:122] "RemoveContainer" containerID="5488a65d5fc99fc6979dda312e2bb17f075c6abffba12961f013ae2cb7a291ba"
Nov 03 08:09:31.172159 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.172112    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"]
Nov 03 08:09:31.177101 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.177084    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:31.235362 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.235307    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"]
Nov 03 08:09:31.259628 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.259581    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx5g8\" (UniqueName: \"kubernetes.io/projected/143a4c0b-7849-470c-946d-bd54fb4b690b-kube-api-access-kx5g8\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:31.259820 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.259642    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:31.259820 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.259757    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:31.283096 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.283064    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:09:31.283265 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.283155    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:09:31.283265 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.283190    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:09:31.283346 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.283273    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55sgn\""
Nov 03 08:09:31.361340 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.361308    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:31.361526 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.361432    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx5g8\" (UniqueName: \"kubernetes.io/projected/143a4c0b-7849-470c-946d-bd54fb4b690b-kube-api-access-kx5g8\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:31.361526 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.361460    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:31.362148 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.362059    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:31.363851 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.363826    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:31.425669 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.425605    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx5g8\" (UniqueName: \"kubernetes.io/projected/143a4c0b-7849-470c-946d-bd54fb4b690b-kube-api-access-kx5g8\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:31.489879 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.489850    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:31.863056 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.863033    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"]
Nov 03 08:09:31.864994 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:09:31.864966    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143a4c0b_7849_470c_946d_bd54fb4b690b.slice/crio-4ee133fbb97e63b0f4bd4d9dec3b14f862d413d9e09f361dedd7ea125fe4a2fc WatchSource:0}: Error finding container 4ee133fbb97e63b0f4bd4d9dec3b14f862d413d9e09f361dedd7ea125fe4a2fc: Status 404 returned error can't find the container with id 4ee133fbb97e63b0f4bd4d9dec3b14f862d413d9e09f361dedd7ea125fe4a2fc
Nov 03 08:09:31.921871 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:31.921842    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k" event={"ID":"143a4c0b-7849-470c-946d-bd54fb4b690b","Type":"ContainerStarted","Data":"4ee133fbb97e63b0f4bd4d9dec3b14f862d413d9e09f361dedd7ea125fe4a2fc"}
Nov 03 08:09:32.928594 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:32.928558    4470 generic.go:356] "Generic (PLEG): container finished" podID="143a4c0b-7849-470c-946d-bd54fb4b690b" containerID="f3ed2a130e1372308684f3b1ce0a6012008eb98b73617d6991d3c2b1e292dd3a" exitCode=0
Nov 03 08:09:32.929032 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:32.928645    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k" event={"ID":"143a4c0b-7849-470c-946d-bd54fb4b690b","Type":"ContainerDied","Data":"f3ed2a130e1372308684f3b1ce0a6012008eb98b73617d6991d3c2b1e292dd3a"}
Nov 03 08:09:34.078827 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.078801    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:34.191634 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.191556    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143a4c0b-7849-470c-946d-bd54fb4b690b-kube-api-access-kx5g8\" (UniqueName: \"kubernetes.io/projected/143a4c0b-7849-470c-946d-bd54fb4b690b-kube-api-access-kx5g8\") pod \"143a4c0b-7849-470c-946d-bd54fb4b690b\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") "
Nov 03 08:09:34.191634 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.191632    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-xks-ca\") pod \"143a4c0b-7849-470c-946d-bd54fb4b690b\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") "
Nov 03 08:09:34.191849 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.191728    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-openshift-ca\") pod \"143a4c0b-7849-470c-946d-bd54fb4b690b\" (UID: \"143a4c0b-7849-470c-946d-bd54fb4b690b\") "
Nov 03 08:09:34.192100 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.192075    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-openshift-ca" pod "143a4c0b-7849-470c-946d-bd54fb4b690b" (UID: "143a4c0b-7849-470c-946d-bd54fb4b690b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:34.193849 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.193824    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-xks-ca" pod "143a4c0b-7849-470c-946d-bd54fb4b690b" (UID: "143a4c0b-7849-470c-946d-bd54fb4b690b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:34.193952 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.193878    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143a4c0b-7849-470c-946d-bd54fb4b690b-kube-api-access-kx5g8" pod "143a4c0b-7849-470c-946d-bd54fb4b690b" (UID: "143a4c0b-7849-470c-946d-bd54fb4b690b"). InnerVolumeSpecName "kube-api-access-kx5g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:34.292634 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.292596    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:09:34.292634 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.292631    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143a4c0b-7849-470c-946d-bd54fb4b690b-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:09:34.292844 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.292649    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx5g8\" (UniqueName: \"kubernetes.io/projected/143a4c0b-7849-470c-946d-bd54fb4b690b-kube-api-access-kx5g8\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:09:34.758985 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:09:34.758946    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:09:34.940173 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.940143    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"
Nov 03 08:09:34.940173 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.940155    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k" event={"ID":"143a4c0b-7849-470c-946d-bd54fb4b690b","Type":"ContainerDied","Data":"4ee133fbb97e63b0f4bd4d9dec3b14f862d413d9e09f361dedd7ea125fe4a2fc"}
Nov 03 08:09:34.940382 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:34.940203    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ee133fbb97e63b0f4bd4d9dec3b14f862d413d9e09f361dedd7ea125fe4a2fc"
Nov 03 08:09:42.743651 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:42.743612    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"]
Nov 03 08:09:42.850726 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:42.850674    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"]
Nov 03 08:09:47.744817 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:47.744758    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65" podUID="3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011" containerName="payload-processing" containerID="cri-o://40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68" gracePeriod=25
Nov 03 08:09:47.851844 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:47.851795    4470 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5" podUID="b910a0eb-dca1-4a2f-9e38-4d31fcf5a322" containerName="payload-pre-processing" containerID="cri-o://134815f27889b00f380f39be37c99b37ac58e4b7605e82b28ee89d8a04851627" gracePeriod=25
Nov 03 08:09:48.000188 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:47.999761    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:09:48.008918 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.008889    4470 generic.go:356] "Generic (PLEG): container finished" podID="3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011" containerID="40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68" exitCode=0
Nov 03 08:09:48.009070 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.008951    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"
Nov 03 08:09:48.009070 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.008975    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65" event={"ID":"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011","Type":"ContainerDied","Data":"40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68"}
Nov 03 08:09:48.009070 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.009034    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65" event={"ID":"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011","Type":"ContainerDied","Data":"af473a8ffc723cef8df94ca91011db5dd427ba4bd31d8153be5863643efd3226"}
Nov 03 08:09:48.009070 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.009057    4470 scope.go:122] "RemoveContainer" containerID="40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68"
Nov 03 08:09:48.011057 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.011035    4470 generic.go:356] "Generic (PLEG): container finished" podID="b910a0eb-dca1-4a2f-9e38-4d31fcf5a322" containerID="134815f27889b00f380f39be37c99b37ac58e4b7605e82b28ee89d8a04851627" exitCode=0
Nov 03 08:09:48.011171 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.011107    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5" event={"ID":"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322","Type":"ContainerDied","Data":"134815f27889b00f380f39be37c99b37ac58e4b7605e82b28ee89d8a04851627"}
Nov 03 08:09:48.022775 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.022754    4470 scope.go:122] "RemoveContainer" containerID="40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68"
Nov 03 08:09:48.023070 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:09:48.023053    4470 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68\": container with ID starting with 40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68 not found: ID does not exist" containerID="40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68"
Nov 03 08:09:48.023151 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.023082    4470 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68"} err="failed to get container status \"40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68\": rpc error: code = NotFound desc = could not find container \"40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68\": container with ID starting with 40e624da084e68c377c484788f9b76943a7e96b0f16b571550730063cad11a68 not found: ID does not exist"
Nov 03 08:09:48.106285 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.106262    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:09:48.133364 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.133325    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-kube-api-access-54sg9\" (UniqueName: \"kubernetes.io/projected/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-kube-api-access-54sg9\") pod \"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011\" (UID: \"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011\") "
Nov 03 08:09:48.133547 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.133404    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-plugins-config-volume\") pod \"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011\" (UID: \"3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011\") "
Nov 03 08:09:48.134120 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.134094    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-plugins-config-volume" pod "3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011" (UID: "3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:48.136220 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.136193    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-kube-api-access-54sg9" pod "3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011" (UID: "3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011"). InnerVolumeSpecName "kube-api-access-54sg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:48.234950 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.234918    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-kube-api-access-2fwxm\" (UniqueName: \"kubernetes.io/projected/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-kube-api-access-2fwxm\") pod \"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322\" (UID: \"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322\") "
Nov 03 08:09:48.235172 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.235059    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-plugins-config-volume\") pod \"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322\" (UID: \"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322\") "
Nov 03 08:09:48.235302 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.235279    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54sg9\" (UniqueName: \"kubernetes.io/projected/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-kube-api-access-54sg9\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:09:48.235367 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.235302    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:09:48.235509 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.235485    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-plugins-config-volume" pod "b910a0eb-dca1-4a2f-9e38-4d31fcf5a322" (UID: "b910a0eb-dca1-4a2f-9e38-4d31fcf5a322"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:48.237177 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.237157    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-kube-api-access-2fwxm" pod "b910a0eb-dca1-4a2f-9e38-4d31fcf5a322" (UID: "b910a0eb-dca1-4a2f-9e38-4d31fcf5a322"). InnerVolumeSpecName "kube-api-access-2fwxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:48.336837 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.336802    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fwxm\" (UniqueName: \"kubernetes.io/projected/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-kube-api-access-2fwxm\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:09:48.336837 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.336831    4470 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322-plugins-config-volume\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:09:48.434422 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.434386    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"]
Nov 03 08:09:48.439409 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:48.439387    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65"]
Nov 03 08:09:49.018463 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:49.018373    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5" event={"ID":"b910a0eb-dca1-4a2f-9e38-4d31fcf5a322","Type":"ContainerDied","Data":"77d498b05fa8087ec2c1c6ef84cafeb73f995dcf2b0dde24b4b22be718fd10dc"}
Nov 03 08:09:49.018463 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:49.018395    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"
Nov 03 08:09:49.018463 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:49.018428    4470 scope.go:122] "RemoveContainer" containerID="134815f27889b00f380f39be37c99b37ac58e4b7605e82b28ee89d8a04851627"
Nov 03 08:09:49.190205 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:49.190164    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"]
Nov 03 08:09:49.241162 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:49.241117    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5"]
Nov 03 08:09:49.805019 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:49.804983    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011" path="/var/lib/kubelet/pods/3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011/volumes"
Nov 03 08:09:49.805445 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:49.805427    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b910a0eb-dca1-4a2f-9e38-4d31fcf5a322" path="/var/lib/kubelet/pods/b910a0eb-dca1-4a2f-9e38-4d31fcf5a322/volumes"
Nov 03 08:09:52.639234 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:52.639199    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"]
Nov 03 08:09:52.689737 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:52.689706    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3ef065-9f25k"]
Nov 03 08:09:53.804648 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:09:53.804604    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143a4c0b-7849-470c-946d-bd54fb4b690b" path="/var/lib/kubelet/pods/143a4c0b-7849-470c-946d-bd54fb4b690b/volumes"
Nov 03 08:10:04.759696 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:10:04.759663    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:10:14.055650 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:14.055615    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"]
Nov 03 08:10:14.105638 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:14.105602    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bhqcz"]
Nov 03 08:10:15.612084 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:15.612056    4470 scope.go:122] "RemoveContainer" containerID="d3eacd4b79839b8b34a989b913c3f970efdeb17c605c39aac40de15bcafc317f"
Nov 03 08:10:15.805103 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:15.805073    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb59133-ab85-4c88-bc00-68bd0b487a50" path="/var/lib/kubelet/pods/2fb59133-ab85-4c88-bc00-68bd0b487a50/volumes"
Nov 03 08:10:24.755215 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:24.754253    4470 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:10:24.755215 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:24.754344    4470 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22" podUID="4dfa6c32-f24d-4d1f-96ff-df222dec2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:30.246720 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.246681    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"]
Nov 03 08:10:30.247184 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.247127    4470 cpu_manager.go:414] "containerMap: removing container" podUID="3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011" containerName="payload-processing"
Nov 03 08:10:30.247184 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.247161    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011" containerName="payload-processing"
Nov 03 08:10:30.247184 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.247181    4470 cpu_manager.go:414] "containerMap: removing container" podUID="143a4c0b-7849-470c-946d-bd54fb4b690b" containerName="revoke-keys"
Nov 03 08:10:30.247184 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.247187    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143a4c0b-7849-470c-946d-bd54fb4b690b" containerName="revoke-keys"
Nov 03 08:10:30.247363 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.247214    4470 cpu_manager.go:414] "containerMap: removing container" podUID="b910a0eb-dca1-4a2f-9e38-4d31fcf5a322" containerName="payload-pre-processing"
Nov 03 08:10:30.247363 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.247221    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b910a0eb-dca1-4a2f-9e38-4d31fcf5a322" containerName="payload-pre-processing"
Nov 03 08:10:30.247363 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.247285    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="b910a0eb-dca1-4a2f-9e38-4d31fcf5a322" containerName="payload-pre-processing"
Nov 03 08:10:30.247363 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.247295    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed4cdac-43d6-42a6-9cf7-99f9c2d8a011" containerName="payload-processing"
Nov 03 08:10:30.247363 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.247301    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="143a4c0b-7849-470c-946d-bd54fb4b690b" containerName="revoke-keys"
Nov 03 08:10:30.251949 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.251931    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:30.285655 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.285621    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ggtb8\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:30.285849 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.285812    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ggtb8\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:30.285920 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.285864    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmvzs\" (UniqueName: \"kubernetes.io/projected/d7f3a381-e230-42c9-962f-45ee386a87f9-kube-api-access-dmvzs\") pod \"maas-api-revoke-sub-f03629db-ggtb8\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:30.309455 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.309421    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"]
Nov 03 08:10:30.357485 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.357453    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:10:30.357667 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.357494    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:10:30.357667 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.357494    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55sgn\""
Nov 03 08:10:30.357667 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.357556    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:10:30.386420 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.386390    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ggtb8\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:30.386576 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.386434    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmvzs\" (UniqueName: \"kubernetes.io/projected/d7f3a381-e230-42c9-962f-45ee386a87f9-kube-api-access-dmvzs\") pod \"maas-api-revoke-sub-f03629db-ggtb8\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:30.386576 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.386555    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ggtb8\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:30.387045 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.387024    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ggtb8\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:30.389053 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.389033    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ggtb8\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:30.450589 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.450561    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmvzs\" (UniqueName: \"kubernetes.io/projected/d7f3a381-e230-42c9-962f-45ee386a87f9-kube-api-access-dmvzs\") pod \"maas-api-revoke-sub-f03629db-ggtb8\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:30.564450 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.564423    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:30.941445 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:30.941413    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"]
Nov 03 08:10:30.943050 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:10:30.943021    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f3a381_e230_42c9_962f_45ee386a87f9.slice/crio-ce685216d359b5a6efc9ceebcf15f6d852f63bceb8653933efba15e77783a3da WatchSource:0}: Error finding container ce685216d359b5a6efc9ceebcf15f6d852f63bceb8653933efba15e77783a3da: Status 404 returned error can't find the container with id ce685216d359b5a6efc9ceebcf15f6d852f63bceb8653933efba15e77783a3da
Nov 03 08:10:31.280352 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:31.280320    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8" event={"ID":"d7f3a381-e230-42c9-962f-45ee386a87f9","Type":"ContainerStarted","Data":"e8ce252ae0bc50d05d911c36d3f904af66a5252fe18a74a30e34722b8a8dec85"}
Nov 03 08:10:31.280715 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:31.280369    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8" event={"ID":"d7f3a381-e230-42c9-962f-45ee386a87f9","Type":"ContainerStarted","Data":"ce685216d359b5a6efc9ceebcf15f6d852f63bceb8653933efba15e77783a3da"}
Nov 03 08:10:31.400851 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:31.400799    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8" podStartSLOduration=1.40078435 podStartE2EDuration="1.40078435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:31.40034561 +0000 UTC m=+4283.732714632" watchObservedRunningTime="2026-11-03 08:10:31.40078435 +0000 UTC m=+4283.733153345"
Nov 03 08:10:32.286490 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:32.286451    4470 generic.go:356] "Generic (PLEG): container finished" podID="d7f3a381-e230-42c9-962f-45ee386a87f9" containerID="e8ce252ae0bc50d05d911c36d3f904af66a5252fe18a74a30e34722b8a8dec85" exitCode=0
Nov 03 08:10:32.286924 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:32.286592    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8" event={"ID":"d7f3a381-e230-42c9-962f-45ee386a87f9","Type":"ContainerDied","Data":"e8ce252ae0bc50d05d911c36d3f904af66a5252fe18a74a30e34722b8a8dec85"}
Nov 03 08:10:33.427330 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:33.427310    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:33.517529 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:33.517488    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-xks-ca\") pod \"d7f3a381-e230-42c9-962f-45ee386a87f9\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") "
Nov 03 08:10:33.517739 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:33.517556    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-openshift-ca\") pod \"d7f3a381-e230-42c9-962f-45ee386a87f9\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") "
Nov 03 08:10:33.517739 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:33.517702    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f3a381-e230-42c9-962f-45ee386a87f9-kube-api-access-dmvzs\" (UniqueName: \"kubernetes.io/projected/d7f3a381-e230-42c9-962f-45ee386a87f9-kube-api-access-dmvzs\") pod \"d7f3a381-e230-42c9-962f-45ee386a87f9\" (UID: \"d7f3a381-e230-42c9-962f-45ee386a87f9\") "
Nov 03 08:10:33.517898 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:33.517875    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-openshift-ca" pod "d7f3a381-e230-42c9-962f-45ee386a87f9" (UID: "d7f3a381-e230-42c9-962f-45ee386a87f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:10:33.518032 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:33.518015    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:10:33.519838 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:33.519803    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-xks-ca" pod "d7f3a381-e230-42c9-962f-45ee386a87f9" (UID: "d7f3a381-e230-42c9-962f-45ee386a87f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:10:33.519955 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:33.519936    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f3a381-e230-42c9-962f-45ee386a87f9-kube-api-access-dmvzs" pod "d7f3a381-e230-42c9-962f-45ee386a87f9" (UID: "d7f3a381-e230-42c9-962f-45ee386a87f9"). InnerVolumeSpecName "kube-api-access-dmvzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:10:33.618575 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:33.618532    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmvzs\" (UniqueName: \"kubernetes.io/projected/d7f3a381-e230-42c9-962f-45ee386a87f9-kube-api-access-dmvzs\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:10:33.618575 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:33.618570    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3a381-e230-42c9-962f-45ee386a87f9-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:10:34.297224 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:34.297194    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8"
Nov 03 08:10:34.297414 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:34.297223    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ggtb8" event={"ID":"d7f3a381-e230-42c9-962f-45ee386a87f9","Type":"ContainerDied","Data":"ce685216d359b5a6efc9ceebcf15f6d852f63bceb8653933efba15e77783a3da"}
Nov 03 08:10:34.297414 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:34.297272    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce685216d359b5a6efc9ceebcf15f6d852f63bceb8653933efba15e77783a3da"
Nov 03 08:10:34.758829 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:10:34.758782    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:10:50.083288 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:50.083206    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"]
Nov 03 08:10:50.086520 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:50.086491    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xtm5j"]
Nov 03 08:10:51.097372 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.097330    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"]
Nov 03 08:10:51.098021 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.098000    4470 cpu_manager.go:414] "containerMap: removing container" podUID="d7f3a381-e230-42c9-962f-45ee386a87f9" containerName="revoke-keys"
Nov 03 08:10:51.098066 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.098025    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f3a381-e230-42c9-962f-45ee386a87f9" containerName="revoke-keys"
Nov 03 08:10:51.098204 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.098191    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f3a381-e230-42c9-962f-45ee386a87f9" containerName="revoke-keys"
Nov 03 08:10:51.101806 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.101788    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:51.158547 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.158513    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"]
Nov 03 08:10:51.206822 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.206774    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:10:51.206822 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.206818    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:10:51.207078 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.206882    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:10:51.207078 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.207060    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55sgn\""
Nov 03 08:10:51.247477 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.247448    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-9zph9\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:51.247671 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.247578    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-9zph9\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:51.247747 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.247686    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rlgc\" (UniqueName: \"kubernetes.io/projected/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-kube-api-access-8rlgc\") pod \"maas-api-revoke-sub-0165d7d2-9zph9\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:51.348681 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.348590    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-9zph9\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:51.348867 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.348685    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rlgc\" (UniqueName: \"kubernetes.io/projected/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-kube-api-access-8rlgc\") pod \"maas-api-revoke-sub-0165d7d2-9zph9\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:51.348867 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.348729    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-9zph9\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:51.349419 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.349395    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-9zph9\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:51.351231 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.351210    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-9zph9\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:51.413266 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.413235    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rlgc\" (UniqueName: \"kubernetes.io/projected/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-kube-api-access-8rlgc\") pod \"maas-api-revoke-sub-0165d7d2-9zph9\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:51.415213 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.415193    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:51.791281 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.791245    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"]
Nov 03 08:10:51.793675 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:10:51.793643    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc758b1b8_4ce6_48ae_b6d7_007b4c6edb2b.slice/crio-790700b0673ac385863613c191c45d03c9d6c9f7c57196021ebec0706ef71bb7 WatchSource:0}: Error finding container 790700b0673ac385863613c191c45d03c9d6c9f7c57196021ebec0706ef71bb7: Status 404 returned error can't find the container with id 790700b0673ac385863613c191c45d03c9d6c9f7c57196021ebec0706ef71bb7
Nov 03 08:10:51.807243 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:51.807218    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea763a73-892b-4833-b03e-6a433cd39931" path="/var/lib/kubelet/pods/ea763a73-892b-4833-b03e-6a433cd39931/volumes"
Nov 03 08:10:52.424229 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:52.424190    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9" event={"ID":"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b","Type":"ContainerStarted","Data":"2a4fea372419b88478f4097bffa14c93e9924f43567b0e03bc39fea63dc3bd3f"}
Nov 03 08:10:52.424743 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:52.424243    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9" event={"ID":"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b","Type":"ContainerStarted","Data":"790700b0673ac385863613c191c45d03c9d6c9f7c57196021ebec0706ef71bb7"}
Nov 03 08:10:52.547032 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:52.546970    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9" podStartSLOduration=1.5469568009999999 podStartE2EDuration="1.546956801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:52.545205098 +0000 UTC m=+4304.877574093" watchObservedRunningTime="2026-11-03 08:10:52.546956801 +0000 UTC m=+4304.879325795"
Nov 03 08:10:53.433546 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:53.433508    4470 generic.go:356] "Generic (PLEG): container finished" podID="c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b" containerID="2a4fea372419b88478f4097bffa14c93e9924f43567b0e03bc39fea63dc3bd3f" exitCode=0
Nov 03 08:10:53.434022 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:53.433593    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9" event={"ID":"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b","Type":"ContainerDied","Data":"2a4fea372419b88478f4097bffa14c93e9924f43567b0e03bc39fea63dc3bd3f"}
Nov 03 08:10:54.576552 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:54.576527    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:54.681104 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:54.681065    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-kube-api-access-8rlgc\" (UniqueName: \"kubernetes.io/projected/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-kube-api-access-8rlgc\") pod \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") "
Nov 03 08:10:54.681353 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:54.681119    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-openshift-ca\") pod \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") "
Nov 03 08:10:54.681353 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:54.681177    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-xks-ca\") pod \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\" (UID: \"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b\") "
Nov 03 08:10:54.681590 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:54.681564    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-openshift-ca" pod "c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b" (UID: "c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:10:54.683461 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:54.683438    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-xks-ca" pod "c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b" (UID: "c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:10:54.683569 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:54.683499    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-kube-api-access-8rlgc" pod "c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b" (UID: "c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b"). InnerVolumeSpecName "kube-api-access-8rlgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:10:54.782557 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:54.782516    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rlgc\" (UniqueName: \"kubernetes.io/projected/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-kube-api-access-8rlgc\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:10:54.782557 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:54.782550    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:10:54.782557 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:54.782560    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:10:55.445030 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:55.444997    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9"
Nov 03 08:10:55.445030 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:55.445014    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9zph9" event={"ID":"c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b","Type":"ContainerDied","Data":"790700b0673ac385863613c191c45d03c9d6c9f7c57196021ebec0706ef71bb7"}
Nov 03 08:10:55.445291 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:10:55.445060    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="790700b0673ac385863613c191c45d03c9d6c9f7c57196021ebec0706ef71bb7"
Nov 03 08:11:02.995528 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:02.995490    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"]
Nov 03 08:11:02.996501 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:02.996474    4470 cpu_manager.go:414] "containerMap: removing container" podUID="c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b" containerName="revoke-keys"
Nov 03 08:11:02.996501 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:02.996496    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b" containerName="revoke-keys"
Nov 03 08:11:02.996652 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:02.996568    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b" containerName="revoke-keys"
Nov 03 08:11:03.002555 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.002532    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:03.062647 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.062605    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"]
Nov 03 08:11:03.065471 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.065443    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ts98v\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:03.065630 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.065482    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmpb\" (UniqueName: \"kubernetes.io/projected/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-kube-api-access-vvmpb\") pod \"maas-api-revoke-sub-6b19338f-ts98v\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:03.065630 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.065514    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ts98v\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:03.110562 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.110515    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:11:03.110562 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.110538    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55sgn\""
Nov 03 08:11:03.110797 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.110626    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:11:03.111162 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.111110    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:11:03.166741 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.166702    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ts98v\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:03.166943 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.166753    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmpb\" (UniqueName: \"kubernetes.io/projected/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-kube-api-access-vvmpb\") pod \"maas-api-revoke-sub-6b19338f-ts98v\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:03.166943 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.166785    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ts98v\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:03.167536 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.167515    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ts98v\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:03.169245 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.169229    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ts98v\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:03.234714 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.234678    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmpb\" (UniqueName: \"kubernetes.io/projected/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-kube-api-access-vvmpb\") pod \"maas-api-revoke-sub-6b19338f-ts98v\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:03.316068 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.316037    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:03.689897 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:03.689872    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"]
Nov 03 08:11:03.691669 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:11:03.691642    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5a11d4_1a52_474c_9696_5adb3b5a4b48.slice/crio-abf9e76aa643f47b0e43c32e4274c9f200e9ac91ec1f1ea643a57fe3009fb6c0 WatchSource:0}: Error finding container abf9e76aa643f47b0e43c32e4274c9f200e9ac91ec1f1ea643a57fe3009fb6c0: Status 404 returned error can't find the container with id abf9e76aa643f47b0e43c32e4274c9f200e9ac91ec1f1ea643a57fe3009fb6c0
Nov 03 08:11:04.493468 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:04.493422    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v" event={"ID":"ee5a11d4-1a52-474c-9696-5adb3b5a4b48","Type":"ContainerStarted","Data":"5abd526bed1b14cbf9d8b51e2d3a3dadb58e5e46f8741fa5c77136c135824fcf"}
Nov 03 08:11:04.493863 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:04.493488    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v" event={"ID":"ee5a11d4-1a52-474c-9696-5adb3b5a4b48","Type":"ContainerStarted","Data":"abf9e76aa643f47b0e43c32e4274c9f200e9ac91ec1f1ea643a57fe3009fb6c0"}
Nov 03 08:11:04.614297 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:04.614239    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v" podStartSLOduration=2.614223479 podStartE2EDuration="2.614223479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:04.612174514 +0000 UTC m=+4316.944543535" watchObservedRunningTime="2026-11-03 08:11:04.614223479 +0000 UTC m=+4316.946592472"
Nov 03 08:11:04.759330 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:11:04.759246    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:11:05.500096 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:05.500061    4470 generic.go:356] "Generic (PLEG): container finished" podID="ee5a11d4-1a52-474c-9696-5adb3b5a4b48" containerID="5abd526bed1b14cbf9d8b51e2d3a3dadb58e5e46f8741fa5c77136c135824fcf" exitCode=0
Nov 03 08:11:05.500528 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:05.500149    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v" event={"ID":"ee5a11d4-1a52-474c-9696-5adb3b5a4b48","Type":"ContainerDied","Data":"5abd526bed1b14cbf9d8b51e2d3a3dadb58e5e46f8741fa5c77136c135824fcf"}
Nov 03 08:11:06.642476 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:06.642448    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:06.704338 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:06.704297    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-xks-ca\") pod \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") "
Nov 03 08:11:06.704540 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:06.704423    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-openshift-ca\") pod \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") "
Nov 03 08:11:06.704540 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:06.704462    4470 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-kube-api-access-vvmpb\" (UniqueName: \"kubernetes.io/projected/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-kube-api-access-vvmpb\") pod \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\" (UID: \"ee5a11d4-1a52-474c-9696-5adb3b5a4b48\") "
Nov 03 08:11:06.704882 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:06.704846    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-openshift-ca" pod "ee5a11d4-1a52-474c-9696-5adb3b5a4b48" (UID: "ee5a11d4-1a52-474c-9696-5adb3b5a4b48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:11:06.706662 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:06.706639    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-xks-ca" pod "ee5a11d4-1a52-474c-9696-5adb3b5a4b48" (UID: "ee5a11d4-1a52-474c-9696-5adb3b5a4b48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:11:06.706791 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:06.706737    4470 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-kube-api-access-vvmpb" pod "ee5a11d4-1a52-474c-9696-5adb3b5a4b48" (UID: "ee5a11d4-1a52-474c-9696-5adb3b5a4b48"). InnerVolumeSpecName "kube-api-access-vvmpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:11:06.806221 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:06.806111    4470 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmpb\" (UniqueName: \"kubernetes.io/projected/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-kube-api-access-vvmpb\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:11:06.806221 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:06.806165    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-xks-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:11:06.806221 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:06.806177    4470 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5a11d4-1a52-474c-9696-5adb3b5a4b48-maas-api-openshift-ca\") on node \"ip-10-0-129-24.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:11:07.510896 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:07.510867    4470 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v"
Nov 03 08:11:07.511067 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:07.510868    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ts98v" event={"ID":"ee5a11d4-1a52-474c-9696-5adb3b5a4b48","Type":"ContainerDied","Data":"abf9e76aa643f47b0e43c32e4274c9f200e9ac91ec1f1ea643a57fe3009fb6c0"}
Nov 03 08:11:07.511067 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:07.511019    4470 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf9e76aa643f47b0e43c32e4274c9f200e9ac91ec1f1ea643a57fe3009fb6c0"
Nov 03 08:11:15.639085 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:15.639056    4470 scope.go:122] "RemoveContainer" containerID="8f4a5e43a50e9f7dd0aa9dc9af699ffca163816ef9ffedb1b7e3ba45f38c10e9"
Nov 03 08:11:23.057655 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:23.057621    4470 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"]
Nov 03 08:11:23.108178 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:23.108127    4470 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b06fd92-jx9qr"]
Nov 03 08:11:23.805469 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:23.805437    4470 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21fe5435-c199-4819-b31d-b38afe394613" path="/var/lib/kubelet/pods/21fe5435-c199-4819-b31d-b38afe394613/volumes"
Nov 03 08:11:26.658400 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:26.658365    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f4dc97bb5-f6qq8_95e7054e-d436-4502-913d-bac856b822e9/manager/0.log"
Nov 03 08:11:27.042964 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:27.042890    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pj94l_f544e17f-b5e1-42d4-9e4a-fd50df088f26/manager/0.log"
Nov 03 08:11:30.039425 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:30.039392    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-9zph9_c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b/revoke-keys/0.log"
Nov 03 08:11:30.269686 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:30.269658    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ts98v_ee5a11d4-1a52-474c-9696-5adb3b5a4b48/revoke-keys/0.log"
Nov 03 08:11:30.507123 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:30.507096    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-ggtb8_d7f3a381-e230-42c9-962f-45ee386a87f9/revoke-keys/0.log"
Nov 03 08:11:34.758647 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:11:34.758603    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:11:35.596094 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:35.596054    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86b9d545b-jsxrj_217855d1-2dcc-476d-8b22-0fc92ff7f242/kube-auth-proxy/0.log"
Nov 03 08:11:36.053995 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:36.053926    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-7bv22_4dfa6c32-f24d-4d1f-96ff-df222dec2a7e/istio-proxy/0.log"
Nov 03 08:11:36.506720 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:36.506694    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-kx6fw_cd812ea7-1e46-432d-acf9-ff1081a5863f/istio-proxy/0.log"
Nov 03 08:11:36.749201 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:36.749174    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-mrk4m_63e33006-3dbb-4d05-a871-af997921a1c2/istio-proxy/0.log"
Nov 03 08:11:36.986513 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:36.986482    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-pmhtb_f9b225a1-0e3c-48c9-9997-46b62edb5072/istio-proxy/0.log"
Nov 03 08:11:38.408187 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:38.408159    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-gxxn9_225e5554-6b09-4cb8-92c4-0a0b8c02f739/payload-pre-processing/0.log"
Nov 03 08:11:40.236116 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:40.236090    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-ffxdf_bdedcd4a-7850-416b-906c-8bffd4b5a2ef/main/0.log"
Nov 03 08:11:40.471528 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:40.471502    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-8mkrz_506a0cd0-0eb3-49d8-b5bf-8c841e823d10/main/0.log"
Nov 03 08:11:40.705920 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:40.705893    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm_69402d62-2b20-4b8b-b0e8-ba113180fcf7/main/0.log"
Nov 03 08:11:41.403434 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:41.403403    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc_ff22aa3b-6c14-43ef-8658-1d6a6d13c53c/main/0.log"
Nov 03 08:11:52.611579 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:52.611499    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f4dc97bb5-f6qq8_95e7054e-d436-4502-913d-bac856b822e9/manager/0.log"
Nov 03 08:11:53.029388 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:53.029287    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pj94l_f544e17f-b5e1-42d4-9e4a-fd50df088f26/manager/0.log"
Nov 03 08:11:55.828086 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:55.828050    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-9zph9_c758b1b8-4ce6-48ae-b6d7-007b4c6edb2b/revoke-keys/0.log"
Nov 03 08:11:56.103397 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:56.103319    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ts98v_ee5a11d4-1a52-474c-9696-5adb3b5a4b48/revoke-keys/0.log"
Nov 03 08:11:56.342317 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:11:56.342287    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-ggtb8_d7f3a381-e230-42c9-962f-45ee386a87f9/revoke-keys/0.log"
Nov 03 08:12:00.843284 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:12:00.843246    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86b9d545b-jsxrj_217855d1-2dcc-476d-8b22-0fc92ff7f242/kube-auth-proxy/0.log"
Nov 03 08:12:01.295305 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:12:01.295178    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-7bv22_4dfa6c32-f24d-4d1f-96ff-df222dec2a7e/istio-proxy/0.log"
Nov 03 08:12:01.757963 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:12:01.757936    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-kx6fw_cd812ea7-1e46-432d-acf9-ff1081a5863f/istio-proxy/0.log"
Nov 03 08:12:01.993090 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:12:01.993054    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-mrk4m_63e33006-3dbb-4d05-a871-af997921a1c2/istio-proxy/0.log"
Nov 03 08:12:02.236847 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:12:02.236816    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-pmhtb_f9b225a1-0e3c-48c9-9997-46b62edb5072/istio-proxy/0.log"
Nov 03 08:12:03.629551 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:12:03.629516    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-gxxn9_225e5554-6b09-4cb8-92c4-0a0b8c02f739/payload-pre-processing/0.log"
Nov 03 08:12:04.759616 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:12:04.759571    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:12:05.197894 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:12:05.197848    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-ffxdf_bdedcd4a-7850-416b-906c-8bffd4b5a2ef/main/0.log"
Nov 03 08:12:05.494795 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:12:05.494717    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-8mkrz_506a0cd0-0eb3-49d8-b5bf-8c841e823d10/main/0.log"
Nov 03 08:12:05.764086 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:12:05.764014    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t25fm_69402d62-2b20-4b8b-b0e8-ba113180fcf7/main/0.log"
Nov 03 08:12:06.588106 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:12:06.588080    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-7fsqc_ff22aa3b-6c14-43ef-8658-1d6a6d13c53c/main/0.log"
Nov 03 08:12:15.660555 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:12:15.660522    4470 scope.go:122] "RemoveContainer" containerID="a7b6470fdf279e6ac27ba2426ac0ce65171b31fd2fb83f6c891e1e1697163e5e"
Nov 03 08:12:34.759015 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:12:34.758980    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:13:04.758740 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:13:04.758702    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:13:15.680795 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:15.680762    4470 scope.go:122] "RemoveContainer" containerID="0c3720d1362f153697a3ac45269b1c8c66e235b65aa3cd62bc982bd0ff9a2a61"
Nov 03 08:13:18.149070 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.149035    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9lpwd/must-gather-nq9np"]
Nov 03 08:13:18.149480 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.149457    4470 cpu_manager.go:414] "containerMap: removing container" podUID="ee5a11d4-1a52-474c-9696-5adb3b5a4b48" containerName="revoke-keys"
Nov 03 08:13:18.149480 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.149469    4470 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5a11d4-1a52-474c-9696-5adb3b5a4b48" containerName="revoke-keys"
Nov 03 08:13:18.149561 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.149551    4470 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5a11d4-1a52-474c-9696-5adb3b5a4b48" containerName="revoke-keys"
Nov 03 08:13:18.152715 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.152698    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lpwd/must-gather-nq9np"
Nov 03 08:13:18.207866 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.207831    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lpwd\"/\"openshift-service-ca.crt\""
Nov 03 08:13:18.257579 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.257545    4470 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lpwd\"/\"kube-root-ca.crt\""
Nov 03 08:13:18.257781 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.257684    4470 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9lpwd\"/\"default-dockercfg-blmhb\""
Nov 03 08:13:18.259806 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.259786    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lpwd/must-gather-nq9np"]
Nov 03 08:13:18.285967 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.285928    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6qd\" (UniqueName: \"kubernetes.io/projected/8c4b5031-4e70-4426-899b-36c4c433586f-kube-api-access-qf6qd\") pod \"must-gather-nq9np\" (UID: \"8c4b5031-4e70-4426-899b-36c4c433586f\") " pod="openshift-must-gather-9lpwd/must-gather-nq9np"
Nov 03 08:13:18.286210 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.286127    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8c4b5031-4e70-4426-899b-36c4c433586f-must-gather-output\") pod \"must-gather-nq9np\" (UID: \"8c4b5031-4e70-4426-899b-36c4c433586f\") " pod="openshift-must-gather-9lpwd/must-gather-nq9np"
Nov 03 08:13:18.387045 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.387010    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8c4b5031-4e70-4426-899b-36c4c433586f-must-gather-output\") pod \"must-gather-nq9np\" (UID: \"8c4b5031-4e70-4426-899b-36c4c433586f\") " pod="openshift-must-gather-9lpwd/must-gather-nq9np"
Nov 03 08:13:18.387281 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.387063    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6qd\" (UniqueName: \"kubernetes.io/projected/8c4b5031-4e70-4426-899b-36c4c433586f-kube-api-access-qf6qd\") pod \"must-gather-nq9np\" (UID: \"8c4b5031-4e70-4426-899b-36c4c433586f\") " pod="openshift-must-gather-9lpwd/must-gather-nq9np"
Nov 03 08:13:18.387388 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.387369    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8c4b5031-4e70-4426-899b-36c4c433586f-must-gather-output\") pod \"must-gather-nq9np\" (UID: \"8c4b5031-4e70-4426-899b-36c4c433586f\") " pod="openshift-must-gather-9lpwd/must-gather-nq9np"
Nov 03 08:13:18.451522 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.451419    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6qd\" (UniqueName: \"kubernetes.io/projected/8c4b5031-4e70-4426-899b-36c4c433586f-kube-api-access-qf6qd\") pod \"must-gather-nq9np\" (UID: \"8c4b5031-4e70-4426-899b-36c4c433586f\") " pod="openshift-must-gather-9lpwd/must-gather-nq9np"
Nov 03 08:13:18.466851 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.466824    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lpwd/must-gather-nq9np"
Nov 03 08:13:18.842284 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.842260    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lpwd/must-gather-nq9np"]
Nov 03 08:13:18.844503 ip-10-0-129-24 kubenswrapper[4470]: W1103 08:13:18.844474    4470 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c4b5031_4e70_4426_899b_36c4c433586f.slice/crio-0ebfac204c160d3fb924c1846b25f431fb673685a33573be9651342dd9f5a6cc WatchSource:0}: Error finding container 0ebfac204c160d3fb924c1846b25f431fb673685a33573be9651342dd9f5a6cc: Status 404 returned error can't find the container with id 0ebfac204c160d3fb924c1846b25f431fb673685a33573be9651342dd9f5a6cc
Nov 03 08:13:18.846284 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:18.846266    4470 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:13:19.203599 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:19.203508    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lpwd/must-gather-nq9np" event={"ID":"8c4b5031-4e70-4426-899b-36c4c433586f","Type":"ContainerStarted","Data":"0ebfac204c160d3fb924c1846b25f431fb673685a33573be9651342dd9f5a6cc"}
Nov 03 08:13:20.212287 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:20.212232    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lpwd/must-gather-nq9np" event={"ID":"8c4b5031-4e70-4426-899b-36c4c433586f","Type":"ContainerStarted","Data":"a795a046ff5d4d14b3e99cae1892490fc820c0b454b3efad5fbb9a7f56bacac6"}
Nov 03 08:13:21.220933 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:21.220897    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lpwd/must-gather-nq9np" event={"ID":"8c4b5031-4e70-4426-899b-36c4c433586f","Type":"ContainerStarted","Data":"e0ba58576bd95126fc7cf8b63d62eba5d90b9d541d49cd595c804d8f714ede83"}
Nov 03 08:13:21.339822 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:21.339764    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9lpwd/must-gather-nq9np" podStartSLOduration=2.224043891 podStartE2EDuration="3.339745814s" totalImagesPullingTime="1.115701923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:21.33951715 +0000 UTC m=+4453.671886146" watchObservedRunningTime="2026-11-03 08:13:21.339745814 +0000 UTC m=+4453.672114809"
Nov 03 08:13:27.692369 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:27.692334    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pw94d_cf458915-e617-486b-8b9f-9019d09ca398/global-pull-secret-syncer/0.log"
Nov 03 08:13:30.680731 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:30.680699    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9v5rw_b5aebe2f-412e-4d2a-b644-7fd13bb62e07/konnectivity-agent/0.log"
Nov 03 08:13:31.499417 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:31.499334    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-24.us-west-1.compute.internal_2e6f5344ea2747ae378e95b2b7a4fbd6/haproxy/0.log"
Nov 03 08:13:34.765560 ip-10-0-129-24 kubenswrapper[4470]: E1103 08:13:34.765521    4470 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vprr7" containerName="extractor"
Nov 03 08:13:37.382645 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.382605    4470 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"]
Nov 03 08:13:37.390449 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.390423    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.404270 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.404243    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-lib-modules\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.404419 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.404278    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-proc\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.404419 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.404300    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-sys\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.404540 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.404436    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqghh\" (UniqueName: \"kubernetes.io/projected/2dce7aaa-45d1-4507-8f53-2de5e21fb678-kube-api-access-vqghh\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.404540 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.404459    4470 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-podres\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.474120 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.474086    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"]
Nov 03 08:13:37.505342 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.505311    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqghh\" (UniqueName: \"kubernetes.io/projected/2dce7aaa-45d1-4507-8f53-2de5e21fb678-kube-api-access-vqghh\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.505342 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.505351    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-podres\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.505600 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.505510    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-lib-modules\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.505600 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.505519    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-podres\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.505600 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.505543    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-proc\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.505600 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.505566    4470 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-sys\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.505600 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.505574    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-lib-modules\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.505876 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.505648    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-sys\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.505876 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.505665    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2dce7aaa-45d1-4507-8f53-2de5e21fb678-proc\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.571454 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.571377    4470 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqghh\" (UniqueName: \"kubernetes.io/projected/2dce7aaa-45d1-4507-8f53-2de5e21fb678-kube-api-access-vqghh\") pod \"perf-node-gather-daemonset-b7nm7\" (UID: \"2dce7aaa-45d1-4507-8f53-2de5e21fb678\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:37.708703 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:37.708615    4470 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:38.084286 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:38.084248    4470 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"]
Nov 03 08:13:38.394578 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:38.394489    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7" event={"ID":"2dce7aaa-45d1-4507-8f53-2de5e21fb678","Type":"ContainerStarted","Data":"77059fe30b0914994c2841c1464deff64de3a1043da02572719d37a99b98916b"}
Nov 03 08:13:38.394578 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:38.394555    4470 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7" event={"ID":"2dce7aaa-45d1-4507-8f53-2de5e21fb678","Type":"ContainerStarted","Data":"192ce11085d3df4677e783e2ea4c65088f52c3e8c3edd9e3fd7480342fbab690"}
Nov 03 08:13:38.394996 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:38.394616    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:38.514928 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:38.514852    4470 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7" podStartSLOduration=1.514835539 podStartE2EDuration="1.514835539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:38.513711807 +0000 UTC m=+4470.846080820" watchObservedRunningTime="2026-11-03 08:13:38.514835539 +0000 UTC m=+4470.847204562"
Nov 03 08:13:44.411416 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:44.411384    4470 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-b7nm7"
Nov 03 08:13:44.962170 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:44.962109    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-dfv4l_785921e2-16fb-4b0a-99b2-228afa4995bd/kube-state-metrics/0.log"
Nov 03 08:13:45.375630 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:45.375603    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-dfv4l_785921e2-16fb-4b0a-99b2-228afa4995bd/kube-rbac-proxy-main/0.log"
Nov 03 08:13:45.796317 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:45.796214    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-dfv4l_785921e2-16fb-4b0a-99b2-228afa4995bd/kube-rbac-proxy-self/0.log"
Nov 03 08:13:46.216918 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:46.216894    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-55b544b886-7zvwx_1aa8ebc8-e1c5-4fb6-b8fa-02aa5cd45a86/metrics-server/0.log"
Nov 03 08:13:46.642679 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:46.642646    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-9hp59_be64ab26-6807-4c79-8008-5c565c77a335/monitoring-plugin/0.log"
Nov 03 08:13:48.342562 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:48.342528    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rn5sc_65a796fc-52a9-4632-8de1-e68bcc3834b4/node-exporter/0.log"
Nov 03 08:13:48.764587 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:48.764516    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rn5sc_65a796fc-52a9-4632-8de1-e68bcc3834b4/kube-rbac-proxy/0.log"
Nov 03 08:13:49.189463 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:49.189435    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rn5sc_65a796fc-52a9-4632-8de1-e68bcc3834b4/init-textfile/0.log"
Nov 03 08:13:52.164032 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:52.164001    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_982afad1-febf-46f3-b82d-ce986908849e/prometheus/0.log"
Nov 03 08:13:52.585674 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:52.585646    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_982afad1-febf-46f3-b82d-ce986908849e/config-reloader/0.log"
Nov 03 08:13:53.010463 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:53.010376    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_982afad1-febf-46f3-b82d-ce986908849e/thanos-sidecar/0.log"
Nov 03 08:13:53.435441 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:53.435406    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_982afad1-febf-46f3-b82d-ce986908849e/kube-rbac-proxy-web/0.log"
Nov 03 08:13:53.860897 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:53.860827    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_982afad1-febf-46f3-b82d-ce986908849e/kube-rbac-proxy/0.log"
Nov 03 08:13:54.287239 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:54.287114    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_982afad1-febf-46f3-b82d-ce986908849e/kube-rbac-proxy-thanos/0.log"
Nov 03 08:13:54.705860 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:54.705833    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_982afad1-febf-46f3-b82d-ce986908849e/init-config-reloader/0.log"
Nov 03 08:13:56.001290 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:56.001250    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-fsmkh_77aba703-0ffc-47c4-b324-c9fd2247c90c/prometheus-operator-admission-webhook/0.log"
Nov 03 08:13:57.694767 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:57.694737    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d4997f974-cg8jj_440f457a-b678-4b70-b5cd-07d53dbbef71/thanos-query/0.log"
Nov 03 08:13:58.115094 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:58.115069    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d4997f974-cg8jj_440f457a-b678-4b70-b5cd-07d53dbbef71/kube-rbac-proxy-web/0.log"
Nov 03 08:13:58.536247 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:58.536166    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d4997f974-cg8jj_440f457a-b678-4b70-b5cd-07d53dbbef71/kube-rbac-proxy/0.log"
Nov 03 08:13:58.966449 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:58.966421    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d4997f974-cg8jj_440f457a-b678-4b70-b5cd-07d53dbbef71/prom-label-proxy/0.log"
Nov 03 08:13:59.395466 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:59.394977    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d4997f974-cg8jj_440f457a-b678-4b70-b5cd-07d53dbbef71/kube-rbac-proxy-rules/0.log"
Nov 03 08:13:59.816423 ip-10-0-129-24 kubenswrapper[4470]: I1103 08:13:59.816393    4470 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d4997f974-cg8jj_440f457a-b678-4b70-b5cd-07d53dbbef71/kube-rbac-proxy-metrics/0.log"